PM Gym

A/B Testing

Let real users, not opinions, settle the debate

An A/B test splits traffic between two versions and measures which one performs better on a metric you chose in advance. Simple in concept, easy to get wrong in practice — this guide covers how to design a test you can trust and read its results honestly.

Step-by-step lessons

Run a Test You Can Trust

Four short lessons: what A/B testing is and when to use it, designing a valid test, statistical significance and common pitfalls, and reading results.

1

What A/B Testing Is (and Isn't For)

An A/B test randomly splits traffic between a control (A, the current version) and a variant (B, the change), then compares a metric chosen in advance. It's a precise tool — and precise tools have a scope.

Good fit

Isolated, well-defined change

Enough traffic, a clear single metric — a button, a flow, a headline.

Poor fit

Big strategic bets, tiny traffic, "why" questions

A/B testing tells you which option wins, not why, and needs real volume to work.

Quick check

A team wants to A/B test whether to pivot the entire company to a new business model. Is A/B testing the right tool?

2

Designing a Test You Can Trust

A trustworthy A/B test follows a short checklist, every time.

1

Hypothesis first

What you expect to happen, and why, before you look at any result.

2

One variable changed

Only the thing you're testing differs between control and variant.

3

Randomized split

Users are assigned to A or B by chance, not by any pattern that could bias the groups.

4

Sample size & duration set in advance

Calculated before running, not eyeballed once results start coming in.

Quick check

A team changes both the headline and the button color in their "B" variant, versus the original "A." What's the design flaw?

3

Statistical Significance and Common Pitfalls

Statistical significance is your confidence that an observed difference is real rather than random noise — commonly using a threshold like p < 0.05, which means that if there were truly no difference, you'd see a gap this large less than 5% of the time by chance. Three pitfalls quietly undermine it.

Pitfall

Peeking

Checking results early and often, stopping the moment it looks significant, inflates false positives.

Pitfall

Novelty Effect

Users react to anything new, temporarily — early lift can fade once the change stops feeling new.

Pitfall

Sample Ratio Mismatch

If the actual split isn't the intended 50/50, something in the setup is likely broken.

Quick check

A team checks their test results every morning and stops the test the moment it crosses statistical significance on day 3 of a planned 4-week test. What's the risk?

4

Reading and Acting on Results

A finished test still needs careful reading — the top-line number rarely tells the whole story.

A lift that fades

A redesigned homepage shows a 15% signup lift in its first three days, then shrinks to 2% by week three. That's the novelty effect: users reacted to the new look, then settled back to normal behavior. Run tests long enough to see past the spike, and segment results — an aggregate win can hide a loss for a specific user group.

Statistical significance isn't the same as practical significance. A tiny but "real" lift might not be worth the complexity of shipping it.

Quick check

A test result is statistically significant with a 0.3% lift in conversion, but implementing the winning variant permanently would add real long-term maintenance complexity. What should the team weigh alongside statistical significance?

Review the concepts

A/B Testing Flashcards

6 cards covering the essentials. Click a card to flip it.

Foundation Card 1 of 6

A/B Testing

Click to flip

Tip: say the answer out loud before flipping.

Explanation

In practice

1 / 6
Apply what you learned

Practice Scenarios

15 situations that test whether you can design a valid A/B test, spot a statistical pitfall, or read a result honestly.

Scenario 1

A team wants to test a new checkout flow but only gets 50 visitors a week to that page.

What's the concern?

Scenario 2

A team runs a test with three different variants (A, B, and C) simultaneously, splitting traffic three ways.

Is this a valid design?

Scenario 3

A test's control group gets the old checkout flow, and the variant group gets a new checkout flow that also happens to load on a faster server.

What's the design problem?

Scenario 4

A team observes their test's traffic split is landing at 62/38 instead of the intended 50/50.

What should they do?

Scenario 5

A test reaches p < 0.05 after just 200 total visitors, far below the pre-calculated sample size of 5,000.

Should the team trust this early result?

Scenario 6

A redesigned homepage shows a 15% lift in signups during its first three days, then the lift shrinks to 2% by week three.

What's the likely explanation?

Scenario 7

A PM wants to know "why" users prefer variant B over variant A, not just which one performs better.

What should supplement the A/B test?

Scenario 8

A team pre-registers success criteria (95% confidence, minimum 3% lift) before running a test, then hits 95% confidence with only a 1% lift.

What should they do?

Scenario 9

A test is stopped at the pre-planned sample size, but the result shows the variant performing worse than control, with high confidence.

Is this test a failure?

Scenario 10

A team segments their test results and finds the variant won overall, but actually performed worse specifically for mobile users.

What should the team do with this finding?

Scenario 11

A stakeholder wants to run a test on a page that gets huge traffic but insists on ending it after just 6 hours because "we already have enough data."

What's the concern, even with high traffic?

Scenario 12

A team's variant wins with statistical significance, and someone proposes immediately testing five more small tweaks on top of it before locking in the winner.

What's the risk with this plan?

Scenario 13

A test's variant is a completely redesigned page with a new layout, new copy, new images, and a new checkout flow, all shipped as a single test.

What's the core design flaw here?

Scenario 14

A team wants to declare victory on a test that reached significance, but they never decided in advance how long to run it or how large a sample they needed.

What's missing from their process?

Scenario 15

A test shows a statistically significant result, and a PM says "ship it" without checking whether the lift is large enough to justify the engineering cost of maintaining two code paths going forward.

What step did the PM skip?

Lock it in

Guess the Term

Read the clues and name the concept. The fewer clues you need, the more points you score.

Round 1 Score 0
Keep it handy

A/B Testing Quick Reference

The whole topic on one screen.

When A/B Testing Fits

Good fit

Isolated change, enough traffic, one clear metric.

Poor fit

Strategic bets, tiny traffic, or "why" questions.

Designing a Valid Test

1

Hypothesis first

State the expected effect before you look.

2

One variable

Only the tested element differs.

3

Randomized split

No bias in who lands in A or B.

4

Sample size & duration set in advance

Not eyeballed once results appear.

Statistics & Pitfalls

Statistical significance

p < 0.05 is a common threshold.

Peeking

Stopping early inflates false positives.

Novelty effect

Early lift can fade as things stop feeling new.

Sample ratio mismatch

Uneven split signals a broken setup.

Reading Results Honestly

Run long enough

See past the novelty spike.

Segment results

An aggregate win can hide a real loss.

Practical vs. statistical

Is the effect big enough to be worth it?

Notification