Reading an experiment
Size an A/B test from a baseline and the lift worth detecting, turn the per-arm count into days, and read a p-value without peeking your way to a false win.
On this page
An A/B test answers one question — is the variant better than the control by at least this much — and it answers it only if it was sized for that "this much" before it started. Most experiment failures are not statistical; they are a test that could never have seen the effect it was looking for, read daily until noise crossed a line. Sizing is one line of arithmetic, and reading is knowing what the p-value is not.
Three numbers before the test
The feature: in one tenant's help centre, the assistant offers an answer before the customer files a ticket. The control shows the answer as a paragraph; the variant shows it with the source article cited. The metric is the self-serve rate: the share of help-centre visitors who do not go on to file a ticket, 5.0% today.
Three numbers get written down before anyone sees a result. The baseline, 5.0%, from last month's daily-active query with the metric swapped in. The minimum detectable effect, the smallest lift worth shipping for: a 10% relative lift, 5.0% to 5.5%, because at 2,000 visitors a day that is 10 tickets a day the agents never see, about 3,600 a year, which is what makes the citation work worth doing. And the settings: a 5% significance level and 80% power, the conventions almost every team uses. Plainly: a 5% chance of declaring a difference when there is none, and an 80% chance of catching the lift if it is really there.
The second number is the one PMs skip, and it is the one that sizes the test. "Any improvement" is not a detectable effect; the smaller the lift you want to see, the more users it takes, by the square.
Sizing it
Kohavi, Tang and Xu give the rule of thumb for the 5%/80% settings: users per arm, n ≈ 16 × σ² ÷ δ², where σ² is the variance of the metric and δ is the absolute difference to detect. For a rate, σ² = p × (1 − p).
σ² = 0.05 × 0.95 = 0.0475
δ = 0.05 × 0.10 = 0.005
n = 16 × 0.0475 ÷ 0.005² = 16 × 0.0475 ÷ 0.000025 = 30,400 per arm
Call it 30,000. The exact formula gives 31,200; the rule gives 30,400; the baseline moved more than that between last month and this one, so a sample size has two significant figures and no more. Two arms at 30,000 is 60,000 visitors, and at 2,000 a day that is a 30-day test. It is also a calendar fact the roadmap has to carry: the decision arrives in a month, not at the next weekly review.
The square is what the bars show. A 5% lift needs 121,600 per arm, 120 days: untestable on one tenant, which is itself a decision — either accept that only a 10% lift is detectable here or run it across tenants. A 20% lift needs 7,600, eight days. The sample-size exercise turns the rule into a function so the arithmetic is never done in a meeting.
The arithmetic is the same back-of-envelope habit as the capacity estimates in the system-design course: state the inputs, round hard, check against one known number. The known number here is the shape: a 5% baseline and a 10% lift is tens of thousands per arm, always.
What a p-value is not
The readout at day 30: control 5.02%, variant 5.42%, lift 8.0% relative, p = 0.03, 95% interval on the lift from 1% to 15%.
The p-value is not the probability the variant is better. It is not the probability the result is due to chance. It is this: if there were no difference at all, the chance of seeing a gap at least this big is 3%. That is evidence against "no difference" and nothing more. The interval is the more useful line, and it is wide: the true lift is somewhere between 1% and 15%, and a 1% lift is not worth shipping for. A p of 0.03 with an interval that brushes zero is a result to ship cautiously and measure after, not a result to announce.
Three habits follow. Read the interval, not the p. Report the absolute effect in the business unit — 0.4 points of self-serve is 8 tickets a day for this tenant — because a relative lift on a small base sounds larger than it is. And decide the ship rule before the readout: "ship if the interval excludes zero and the point estimate is above 5% relative", written in the experiment doc on day 0.
The mistake: peeking
The team checks the dashboard every morning. On day 9 the variant is at p = 0.04 and someone proposes shipping. The test was sized for 30 days; on day 9 it has 9,000 per arm, less than a third of what the 10% lift needs, so the only lifts it can legitimately see are above about 18% relative, and the point estimate on day 9 is noise with a p-value attached.
Why peeking inflates false positives: each look is another chance for random variation to cross the 0.05 line, and a test stopped at the first crossing has been given thirty chances at a 5% event rather than one. Kohavi, Tang and Xu treat this as one of the standard pitfalls; the usual simulation result for a month of daily looks with a stop at the first p under 0.05 is a false-positive rate of order 20–30%, not 5%. The variant that "won on day 9" is a coin that came up heads early.
The fix is a fixed horizon: the sample size and the read date are in the experiment doc on day 0, the dashboard shows the days remaining rather than the p-value until then, and the result is read once. Teams that need to stop early use a sequential method designed for repeated looks, which adjusts the threshold at each look; the name to ask the data team for is sequential testing, and the thing not to do is apply a fixed-horizon p-value thirty times.
Segments are hypotheses, not results
The day-30 readout also shows the lift by browser, by country, by article category and by hour of day: twenty segments, and in one of them, mobile visitors in one region, the variant loses at p = 0.04. At a 5% level, twenty segments produce one such result by chance on average. A segment that moves against the headline is a hypothesis for the next test, not a reason to hold this one, unless it was named before the test started as a guardrail. Guardrails are read as results because they were pre-registered; everything else in the segment table is a place to look next.
In an interview
The execution question is "the test shows a 12% lift at p = 0.03; do you ship?", and what is being tested is whether you ask for the sample size, the interval and the pre-registered rule before answering. Say: "sized for what effect, over how many days, and were we at the planned size when this was read?" Then read the interval in the business unit. An answer that reaches "ship" without those three questions has not read the experiment.
The mistake that loses points is treating p under 0.05 as the decision — it is one input — and sizing a test for "any improvement". An interviewer who asks "how long would you run it?" wants a baseline, a minimum detectable effect and the arithmetic; "two weeks" with no numbers is a guess dressed as a plan.
Check yourself
1. The baseline is 20% and the lift worth shipping for is 5% relative, at 2,000 users a day. How long is the test?
σ² = 0.2 × 0.8 = 0.16; δ = 0.01; n = 16 × 0.16 ÷ 0.0001 = 25,600 per arm, call it 26,000. Two arms is 52,000 users, 26 days at 2,000 a day. A higher baseline needs fewer users for the same relative lift because σ² grows slower than δ²; say "about four weeks".
2. Day 14 of a 30-day test: the variant is at p = 0.02 with a 15% lift. Engineering wants to ship. Do you?
No. Day 14 is 14,000 per arm against the 30,000 the 10% lift needs; a 15% point estimate at half the sample is inside the range noise produces, and the p-value has been looked at fourteen times. Hold to day 30. The evidence that would let you stop early is a pre-registered sequential rule, which this test does not have.
3. The headline is a 10% lift with the interval 2% to 18%, and one of twenty segments shows the variant losing at p = 0.04. Ship, hold, or extend?
Ship, and name the segment as the first follow-up test. One losing segment in twenty at the 5% level is the expected number by chance; it was not a pre-registered guardrail, so it is a hypothesis. What would change the answer: the segment being a guardrail named on day 0, or a losing segment that is a third of the traffic rather than a sliver.