Data doesn't speak for itself — someone has to ask it the right question. This guide covers how to approach analysis with a clear mindset, which method fits which question, a few frameworks worth knowing, and the biases that quietly wreck good analysis.
Step-by-step lessons
From Data to Decision
Four short lessons: the analysis mindset, quantitative vs. qualitative, common frameworks, and avoiding bias.
1
From Data to Decision
Analysis starts with a question, not a dashboard. Staring at numbers with no goal rarely
reveals anything; a specific question, tied to a decision, focuses the search.
Everyday example — how a doctor works
A doctor doesn't run every possible test on every patient. They form a hypothesis about
what might be wrong, then order the specific tests that would confirm or rule it out.
Good analysis works the same way: start with a question, then go find the data that
answers it.
Quick check
A PM opens the analytics dashboard with no particular question in mind, just "to see what's there." What's the risk of this approach?
2
Quantitative vs. Qualitative Analysis
The two main modes of analysis answer different questions, and the strongest work pairs
both.
What & how much
Quantitative
Numbers at scale. Strong at measuring size and trend, weak at explaining why.
Why
Qualitative
Interviews, recordings, open-ended feedback. Strong at reasoning, weak at telling you how many.
Quant tells you where to look. Qual tells you why it's happening there.
Quick check
A funnel shows a 40% drop-off at the payment step, but the team doesn't know why users are leaving. What should they add to the analysis?
3
A Few Frameworks Worth Knowing
A handful of structured approaches cover most product analysis needs.
Funnel Analysis
Break a multi-step process into stages and find where the biggest drop-off happens.
Cohort Analysis
Group users by a shared starting point (like signup week) and track how their behavior changes over time.
Segmentation
Slice a metric by a meaningful dimension to see if an aggregate number is hiding very different sub-stories.
Quick check
A team's overall retention curve looks flat month over month, but they haven't checked whether newer signup cohorts are retaining better or worse than older ones. What analysis technique would reveal this?
4
Avoiding Bias in Analysis
The same data can support very different, equally confident-sounding conclusions. Three
traps distort analysis most often.
Trap 1
Correlation vs. Causation
Two things moving together doesn't mean one causes the other — a hidden third factor might drive both.
Trap 2
Cherry-Picking
Searching until you find the metric or time window that supports what you already believed.
Trap 3
Simpson's Paradox
A trend in several groups can reverse when the groups are combined, if their sizes differ enough.
Quick check
A report claims "ice cream sales cause more drownings" because both rise together every summer. What's the analytical error?
Review the concepts
Analysis Flashcards
6 cards covering the essentials. Click a card to flip it.
FoundationCard 1 of 6
Analysis Mindset
Click to flip
Tip: say the answer out loud before flipping.
Explanation
In practice
1 / 6
Apply what you learned
Practice Scenarios
15 situations that test whether you can pick the right analysis method, use a framework correctly, or catch a bias distorting a conclusion.
Scenario 1
A PM is asked "how is the product doing?" with no further detail, and immediately starts pulling every metric they can find into one giant report.
What's a better first step?
Scenario 2
A funnel analysis shows the biggest drop-off is between "add to cart" and "enter shipping info," but the team doesn't know if it's a UX problem or a shipping-cost surprise.
What should come next?
Scenario 3
A cohort analysis shows the March 2025 signup cohort retaining much better at 90 days than the January 2025 cohort.
What's a reasonable next step?
Scenario 4
A report says "users on our premium plan are happier, because premium-plan NPS is 20 points higher than free-plan NPS."
What alternative explanation should the analyst consider?
Scenario 5
A dashboard shows "conversion rate" as one aggregate number for both desktop and mobile combined, and it's been flat for months.
What analysis technique might reveal something the aggregate is hiding?
Scenario 6
An analyst notices Hospital A has a higher survival rate than Hospital B for both mild AND severe cases individually, yet Hospital B has a higher overall survival rate.
What's the likely explanation?
Scenario 7
An analyst keeps re-slicing a flat metric by different time windows until they find a 3-day period where it happened to spike, then reports "engagement is up."
What's the analytical problem?
Scenario 8
A PM wants to know why enterprise customers churn at a higher rate than small-business customers.
What analysis approach best answers "why," beyond just confirming the rate difference?
Scenario 9
A team segments a metric by acquisition channel and finds one channel's users convert twice as well as others, then immediately doubles that channel's budget.
What should they check before doing this?
Scenario 10
An analyst reports "feature X caused a 10% revenue increase" based solely on revenue rising in the month after feature X launched.
What's missing from this conclusion?
Scenario 11
A retention chart shows an overall improving trend, but a closer cohort breakdown shows every individual cohort is actually declining.
What's happening?
Scenario 12
A team wants to understand both how many users are affected by a bug and why it's happening to them.
What combination of methods fits best?
Scenario 13
A metric review meeting spends 90% of its time debating a metric that moved by 0.1%, within its normal week-to-week noise range.
What's the issue?
Scenario 14
An analyst wants to compare this quarter's conversion rate to last quarter's, but pricing changed in between the two periods.
What should the analysis account for?
Scenario 15
A team concludes "our new onboarding is a failure" based on one week of data, right after a major unrelated outage occurred during that same week.
What's the analytical error?
Lock it in
Guess the Term
Read the clues and name the concept. The fewer clues you need, the more points you score.