PM Gym

Hypothesis Testing

Turn a guess into something you can prove wrong

Every product decision rests on assumptions. Hypothesis testing makes those assumptions explicit, ranks which ones are riskiest, and designs the cheapest experiment that could prove you wrong — before you spend months building on a guess.

Step-by-step lessons

Test Before You Build

Four short lessons: what a hypothesis is, how to write one well, finding your riskiest assumption, and designing a real test.

1

A Hypothesis Is a Bet You Can Lose

Every roadmap item is really a bet: "if we build this, users will do that." A hypothesis makes that bet explicit and, crucially, falsifiable — stated so a real result could prove it wrong.

Everyday example — the weather forecast

"It might rain" isn't falsifiable — any weather confirms it. "There's an 80% chance of rain by 3pm" is: check the sky at 3pm and you know if the forecast held. A hypothesis needs the same sharpness — a claim specific enough that reality can say yes or no.

Quick check

Which of these is a genuine, falsifiable hypothesis?

2

The Anatomy of a Good Hypothesis

A reliable template keeps hypotheses sharp: "We believe that [doing X] for [this group] will result in [this outcome]. We'll know we're right if we see [this measurable signal]."

The change (X)

Exactly what you're doing differently — one variable, not a bundle of changes.

The group

Who this applies to — all users, or a specific segment.

The expected outcome

What you believe will happen, and roughly by how much.

The measurable signal

The exact metric and threshold that tells you the bet paid off.

Quick check

A team writes: "We believe redesigning the dashboard will make it better." What's missing?

3

Finding the Leap of Faith Assumption

Every hypothesis rests on several smaller assumptions. Some are safe bets; one or two are the leap of faith assumptions — if those are wrong, the whole idea collapses. Test those first.

Test first

High risk + high uncertainty

You're not sure, and being wrong would sink the idea. This is your leap of faith.

Note, don't test yet

High risk, low uncertainty

Matters a lot but you already have strong evidence. Don't waste a test here.

Skip

Low risk either way

Wouldn't sink the idea even if wrong. Testing it burns time you don't need to spend.

Quick check

A marketplace idea assumes: (1) sellers exist in this category, (2) sellers will pay a listing fee, (3) the checkout button should be blue. Which is the leap of faith to test first?

4

Designing a Test You Can Trust

A good test is the cheapest, fastest thing that could actually prove the hypothesis wrong — decided before you run it, not interpreted afterward to fit whatever happened.

Set the bar before you jump

A team runs a landing-page test for a feature idea and gets a 4% signup rate. Is that good? Nobody decided in advance — so half the room calls it a win, half calls it a flop, and the debate has nothing to do with data. Agree "we need 8% to justify building this" beforehand, and the result answers itself.

Define success criteria before you run the test, not after you see the number.

Quick check

Why should success criteria be set before running a test, not after?

Review the concepts

Hypothesis Testing Flashcards

6 cards covering the essentials. Click a card to flip it.

Foundation Card 1 of 6

Hypothesis

Click to flip

Tip: say the answer out loud before flipping.

Explanation

In practice

1 / 6
Apply what you learned

Practice Scenarios

15 situations that test whether you can spot a vague hypothesis, find the riskiest assumption, or catch a test being read after the fact.

Scenario 1

A team says: "Our hypothesis is that redesigning the app will improve things."

What's wrong with this hypothesis?

Scenario 2

A test shows a 3% lift and the team debates for a week whether that counts as a win — no threshold was set beforehand.

What should have prevented this debate?

Scenario 3

An idea rests on: users have smartphones (very likely true), users will pay a $5/month fee (unknown), and onboarding uses blue buttons (trivial).

Which should be tested first?

Scenario 4

A landing-page test hits the pre-agreed 8% signup bar. A stakeholder says: "Let's also check three other metrics we didn't originally care about, and if any looks good, call it a bigger win."

What's the risk here?

Scenario 5

A hypothesis reads: "We believe adding SMS reminders for all users will raise 7-day retention by 15%, measured over 3 weeks."

What makes this a strong hypothesis?

Scenario 6

Two teams ran the "same" experiment but got opposite results — one changed button copy and color at once, the other changed only color.

What principle was violated?

Scenario 7

A PM says: "I already know push notifications will boost engagement, so let's skip testing and build the full feature."

What's the risk of skipping the test?

Scenario 8

Success criteria said "10% lift required." The result comes in at 9.4%. A teammate says: "Close enough, let's call it a win."

What's the concern?

Scenario 9

An assumption map shows one item in "high risk, high uncertainty" and five items in "low risk, low uncertainty."

Where should the next test focus?

Scenario 10

A hypothesis says a redesign will help "users." Data later shows it helped power users but hurt new users.

What does this reveal about the original hypothesis?

Scenario 11

A founder wants to skip writing formal hypotheses because "we move too fast for that."

What's the best response?

Scenario 12

A test's sample was 40 users, mostly from one referral source, and the team wants to greenlight a company-wide rollout.

What's the concern with the test design?

Scenario 13

A pricing hypothesis is deemed "too risky and important to test cheaply," so the team plans a six-month rebuild to test it properly.

Is that the right call?

Scenario 14

After a failed test, a PM says: "The hypothesis was wrong, so this whole idea is dead."

Is that the right conclusion?

Scenario 15

A team wants to test a new pricing model but worries a public test could alienate existing customers if it's visible.

What's a reasonable way to test with lower exposure?

Lock it in

Guess the Term

Read the clues and name the concept. The fewer clues you need, the more points you score.

Round 1 Score 0
Keep it handy

Hypothesis Testing Quick Reference

The whole topic on one screen.

The Hypothesis Template

"We believe [X] for [group] will result in [outcome]. We'll know if [signal]."

Change

One variable, not a bundle.

Group

Who this applies to.

Outcome

What you expect, roughly how much.

Signal

The exact metric and threshold.

Finding the Leap of Faith

High risk + high uncertainty

Test first — this is your leap of faith.

High risk, low uncertainty

Note it, don't waste a test on it.

Low risk either way

Skip — wouldn't sink the idea if wrong.

Designing a Trustworthy Test

One variable at a time

Otherwise you can't tell what caused the result.

Success criteria set in advance

Decide the bar before you see the number.

Representative sample

Small or biased samples don't generalize.

Common Traps

Unfalsifiable wording

"Might help somehow" can never be wrong.

Moving the goalposts

Changing the bar after seeing the result.

Confirmation bias

Reading results the way you wanted them to go.

Notification