Creative testing at volume
Count the false winners in a 50-variant test, discount the winner's lift, and plan a screen-then-confirm test the traffic can actually support.
On this page
A generation tool turns out 50 headline or ad variants in an afternoon. The traffic to test them did not grow with them. Fifty variants each compared with a control, at the usual α = 0.05, will produce winners when none of them works, and the lift reported for any real winner will be too high. The fix is not fewer ideas; it is a test plan that spends traffic where it settles something.
Fifty comparisons, 2.5 false winners
A test at α = 0.05 accepts a 5% chance of a significant result when the variant does nothing. Run one comparison and that is a fair price. Run fifty and the expected count of significant results when nothing works is 50 × 0.05 = 2.5: the false winners.
| Variants tested | False winners expected | Chance of at least one |
|---|---|---|
| 1 | 0.05 | 5% |
| 10 | 0.5 | 40% |
| 20 | 1 | 64% |
| 50 | 2.5 | 92% |
| 100 | 5 | 99% |
The tests are two-sided, so about half of those 2.5 land below control and are quietly dropped. The other half get the budget. The chance that at least one do-nothing variant comes out significantly above control is 1 − 0.975⁵⁰ ≈ 72%. A team that tests 50 variants and gets no significant result in either direction has had an unusual month: the chance is 0.95⁵⁰, under 8%.
The winner's curse
Take the invoicing app from the funnel and AARRR: 100,000 visits a month, 4% of whom sign up. The plan that arrives with the 50 variants is to split a month of traffic across them and the control, about 2,000 visitors each, and ship whatever wins.
At 2,000 visitors an arm the standard error of the difference between a variant and control is √(2 × 0.04 × 0.96 ÷ 2,000) ≈ 0.62 points. A difference is significant at about 1.96 × 0.62 ≈ 1.2 points: a variant has to convert at 5.2% against 4.0%, a 30% relative lift, to be called. Now suppose none of the 50 does anything. Ranked by observed lift, they spread out by chance alone.
The best of 50 shows a lift of about 35% with a true lift of zero. It crosses the line and is shipped, and on full traffic its conversion falls back to 4.0%.
The same selection distorts real winners. Suppose one variant truly lifts sign-ups 10%, from 4.0% to 4.4%. Its true difference is 0.4 points, under one standard error, so at this sample size it is declared a winner only in the runs where noise pushed it past 1.2 points. The lift reported for it is then at least 30%, three times the truth. This is the winner's curse: whatever is picked for being highest was picked partly for its luck, and the luck does not survive rollout. A forecast built on the screen's lift is wrong before it is used.
Traffic is the constraint, not ideas
At the course's test settings, two-sided α = 0.05 and 80% power, detecting a 10% relative lift on a 4% sign-up rate takes about 39,500 visitors in each arm; a 15% lift takes about 18,000. With 100,000 visits a month, a properly sized test at 10% can run control and one challenger a month, with a little left over. Testing all 50 variants that way would need 51 × 39,500 ≈ 2 million visitors, twenty months of traffic. Generation moved the bottleneck from making variants to judging them.
The formula is worked in experiments: holdouts, geo tests and power, whose 4% example gives the same 39,500, and the peeking problem in reading an experiment. The constant from the arithmetic a marketer carries is the one to keep: a 10% lift on a 2% rate takes about 81,000 an arm.
The fixes
Screen, then confirm. Split the job in two. A screen runs many variants cheaply, accepts that it will produce false winners, and has one output: a shortlist. Nothing is shipped from it and no lift from it is reported. A confirmation test then runs the top two or three against control on fresh traffic, sized for the lift worth having. For the top 2 at a 15% lift, with α split between the two comparisons, that is about 21,700 visitors an arm, 65,000 in all, about three weeks of the site's traffic.
Correct the threshold for the number of comparisons. The bluntest correction, Bonferroni, divides α by the number of comparisons: 0.05 ÷ 50 = 0.001, which at 2,000 visitors an arm raises the bar to a 51% lift and blinds the screen. It suits a confirmation stage with two or three comparisons. For screens, the procedure of Benjamini and Hochberg (1995) (opens in a new tab) controls the share of declared winners that are false rather than the chance of any false winner at all, which costs much less power.
Fix the horizon, or test sequentially. Checking results daily and stopping at the first significant reading gives each variant many chances to cross the line, which multiplies the false winners again. Either set the sample size in advance and read once, or use a method built for continuous monitoring, such as the always-valid p-values in Always Valid Inference: Continuous Monitoring of A/B Tests (opens in a new tab) (Johari, Koomen, Pekelis and Walsh, Operations Research, 2022).
Report the confirmation lift. The number that goes into a forecast is the lift measured in the confirmation test, on traffic that played no part in choosing the variant.
Keep a standing holdout. Leave 5% of traffic on one fixed control permanently. Twelve "winning" tests in a year that each claimed 8% compound to 1.08¹² ≈ 2.5 times the sign-up rate. If the live rate is 4.2% against the holdout's 4.0%, the program's claims were mostly the curse, and only the holdout can say so.
The same arithmetic limits personalisation at segment-of-one, where 80,000 variants each meet one reader and none can be tested at all.
In practice
To the creative team, or a tool vendor reporting winners: "How many variants were compared, at what α, and what did the winner do on traffic that did not pick it?" A results page that shows a winning variant's lift without the number of comparisons behind it is not a result yet.
To the team: generate as many variants as you like, screen them on cheap traffic or a cheaper metric, confirm two or three properly, and ship from the confirmation only. Budget the confirmation in visitors before the variants are made, because it is the scarce part.
The mistake that costs money is shipping the top of a 50-variant leaderboard and booking its screen lift in the forecast. With nothing working, the best of 50 shows about 35%; with one real 10% winner, the reported lift is at least three times the truth, and the plan built on it misses by the difference.
Check yourself
1. Your testing tool reports 3 winners from 40 variants at p under 0.05, with lifts of 32%, 29% and 31%. Do you ship all three?
No. 40 × 0.05 = 2 false winners are expected when nothing works, about one of them above control, and lifts clustered just past the significance line are what selected noise looks like. Run the three against control on fresh traffic, sized for the smallest lift worth having, with α split three ways, and expect any confirmed lift to be well under 30%.
2. The site gets 100,000 visits a month at a 4% sign-up rate. How many challengers a month can you confirm at a 10% lift, and what does that mean for 50 generated variants?
About 39,500 visitors an arm, so 100,000 a month covers control and one challenger, about 2.5 arms. Fifty variants cannot each be confirmed; screen them, keep two, and confirm those over about a month and a half, or confirm at 15% in three weeks if a 15% lift is the smallest worth shipping.
3. Last year twelve tests each shipped a winner with a claimed 8% lift, yet the sign-up rate moved from 4.0% to 4.2%. What happened, and what do you change?
Compounded, the claims say 4.0% × 1.08¹² ≈ 10%; the rate rose 5%. Most of the claimed lifts were false winners or winners' curse. Add a standing 5% holdout on a fixed control, report only confirmation-stage lifts, and judge the testing program by the holdout gap rather than by the sum of its claims.