Experiments: holdouts, geo tests and power
Size a lift test from its baseline and the lift that changes the budget, pick the share to hold out, and design a geo test where people cannot be split.
On this page
An experiment measures what attribution cannot: the orders that would not have happened without the ad. Withhold the ads from a random group, compare it with the group that saw them, and the difference is the effect. The method is simple and the size is not, because an ad's effect on any one person is small next to the noise in whether that person buys.
A holdout, and what makes it a test
Split the audience at random before the campaign runs. The exposed group sees the ads as usual; the holdout sees none, or a blank placeholder in the same slot so both groups are counted the same way. Because the split is random, the groups differ only in the ads, and the holdout's conversion rate answers "what happens without them". Incremental orders are the exposed group's orders minus what the holdout's rate predicts for a group that size.
Three conditions make it a test rather than a comparison. The split is made before anyone is exposed. The outcome is counted in your own order data, not in the ad platform's report. And the size and the decision rule are written down before launch. The last is the one teams skip; reading an experiment in the ai-pm course shows what happens to a test that is read every morning until it wins.
The sample size, worked
Every test in this course uses two-sided α = 0.05 and 80% power, the settings fixed in the arithmetic a marketer carries. For conversion rates p₁ and p₂ in the two groups:
n per arm = (z_α/2 + z_β)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₂ − p₁)²
z_α/2 = 1.96 (two-sided 5%) z_β = 0.84 (80% power) (1.96 + 0.84)² = 7.84, 7.85 unrounded
Worked for a 2% baseline and a 10% relative lift, 2.0% to 2.2%:
variances 0.020 × 0.980 + 0.022 × 0.978 = 0.0196 + 0.0215 = 0.0411
gap 0.022 − 0.020 = 0.002, squared 0.000004
n per arm 7.85 × 0.0411 ÷ 0.000004 ≈ 80,700, about 81,000
81,000 in each group, 162,000 in all, to see a lift most marketers would call modest. The ai-pm course sizes the same test with Kohavi, Tang and Xu's rule of thumb, 16 × p(1 − p) ÷ gap², which gives 78,400 here. The two agree within 3%; its sample-size exercise codes the rule, and this module's code exercise adds what a marketing test changes, an unequal split.
Two levers dominate. Halve the lift you need to see and the sample quadruples, because the gap is squared. And at a fixed relative lift the sample grows roughly as 1 ÷ p: a baseline twenty times smaller needs about twenty times the people.
Why ad experiments need millions
A landing page converts 2% of its visitors. An advertising test rarely has a baseline like that, because its unit is everyone the campaign could reach, and most will not buy this month whatever they see. At a 0.1% baseline the same 10% lift needs about 1.65 million people per arm. A yes-or-no outcome at 0.1% has a standard deviation about 32 times its mean, √(0.999 ÷ 0.001) ≈ 31.6, and the ad is trying to move that mean by a tenth.
Randall Lewis and Justin Rao found exactly this in 25 large field experiments with major US retailers and brokerages ("The Unfavorable Economics of Measuring the Returns to Advertising", Quarterly Journal of Economics, 2015). Individual sales were so volatile that a standard deviation ten times the mean was common; the median confidence interval on return on investment was over 100 percentage points wide; and an informative experiment could easily need more than 10 million person-weeks. The conclusion is not that ads fail but that most advertisers cannot tell, and a test too small to see the effect returns "not significant", which gets read as "no effect".
A 5% holdout on a $200k month
A company larger than module 2's invoicing app, with the same default customer, spends $200,000 a month on paid social. At the course's $20 CPM that is 10 million impressions; assume five per person, so 2 million new people a month. The platform credits it with 600 customers, $333 each on the course's chain of 1% clicking, 3% signing up and 20% paying. Assume 0.10% of these people would become customers within 30 days without the ads; if all 600 are real, the exposed group converts at 0.13%, a 30% lift.
Size the test for the decision, not for the claim. At the course's $368 ceiling, a customer's margin over the 12-month payback target rather than its lifetime $800, the channel pays back only if about 543 of the 600 are real. At half the claim, 300, each costs $667, and the budget should shrink. So the test must see a 15% lift, 0.100% to 0.115%. With unequal groups, each group's variance is divided by its share of the audience, so a small holdout is expensive in people:
people = 7.85 × [p₁(1 − p₁) ÷ h + p₂(1 − p₂) ÷ (1 − h)] ÷ gap² h = share held out
h = 50% 1.5 million people about 3 weeks of reach
h = 5% 7.4 million people about 16 weeks
The instinct is the 5% holdout. It withholds ads from one person in twenty, so it forgoes at most 5% of what the channel causes: 30 of the 600 claimed customers a month, about $10,000. Its price is the calendar, sixteen weeks to answer a question the budget review asks monthly. Over the whole test the forgone customers differ far less than the shares do: 370,000 held out × 0.03% ≈ 111 at 5%, against 135 at 20% and 225 at 50%. Pick the share by the date the decision is needed, then accept the cost that date implies. Here, 20% for five weeks.
Geo tests: when people cannot be split
Some channels cannot be randomised person by person: a podcast read, streaming TV, billboards, any channel whose sales land in shops where nobody is identified. And a platform's own lift test counts only the conversions it can match to its users, run by the party being graded. The alternative is to randomise regions: switch the channel off in a random set and compare their sales with the rest, each region against a forecast from its own history. eBay's non-brand search test was one at scale, 30% of its US traffic for 60 days.
The unit is now a region, and there are dozens, not millions. Assume 40 regions, each of whose four-week sales lands within ±5% of its forecast (one standard deviation), and switch the channel off in 8:
standard error 5% × √(1/8 + 1/32) = 5% × 0.395 ≈ 2.0%
smallest effect (1.96 + 0.84) × 2.0% ≈ 5.5% of sales
A channel expected to drive 8% of sales is readable this way; one expected to drive 3% is not without more regions or a tighter forecast. Pairing regions with similar histories and randomising within each pair is the cheapest way to tighten it. Keep reading for two weeks after the channel returns, four for a podcast or video, because an ad's effect lingers, which marketing mix modelling calls adstock. A geo test also saves the dark regions' media (a fifth of a $200,000 channel for a month is $40,000), so when the channel turns out weak the test costs almost nothing.
In practice
To a platform or agency proposing a lift test: "What baseline and what lift is it sized for, how many people end up in the holdout, and do we get the orders in our own data?" A test proposed without the first answer usually comes back "inconclusive": the money spent, the question still open.
To the team, write the rule before launch: "keep the $200,000 if the test shows at least 543 incremental customers a month; below that, cut spend until the cost per incremental customer is back under $368." Then nobody reads the result to fit what they wanted.
The mistake that costs money is running a test too small to see the effect and reading "not significant" as "does not work". A 5% holdout read after four weeks, for a 15% lift that needed sixteen, cuts a channel on noise.
Check yourself
1. A landing page converts 4% and you want to see a 10% lift, 4.0% to 4.4%. Estimate the people per arm, and decide whether a site with 30,000 visitors a week can run it inside a month.
7.85 × (0.04 × 0.96 + 0.044 × 0.956) ÷ 0.004² = 7.85 × 0.0805 ÷ 0.000016 ≈ 39,500 per arm, 79,000 in all: about 2.6 weeks at 30,000 a week. Yes. Doubling the baseline roughly halved the sample.
2. The $200k channel's owner wants a 10% holdout and an answer in four weeks. Do you agree?
No. At 10% the 15% lift needs 3.9 million people, about 8½ weeks at 2 million a month; four weeks reaches about 1.9 million, which sees only a lift of about 22%. Either run 10% for nine weeks, or hold out 30% for four (1.7 million people, 520,000 held out).
3. A podcast is thought to drive 4% of sales. With 40 regions at ±5% and 8 dark, can a geo test see it? If not, what do you change?
No: the smallest visible effect is 5.5%. Twenty dark regions only gets to 5% × √(1/20 + 1/20) × 2.8 ≈ 4.4%. Pair the regions so the forecast error falls to about 3%, and with 20 dark the test sees 3% × 0.316 × 2.8 ≈ 2.7%, under the 4% it needs.