Measurement · Experiments

Design a holdout for a $200k-a-month channel

Choose the unit, share, length and decision rule for a test of $200,000 a month of paid social whose sales partly land in shops, and cost it.

designmediumExercise · 3 min · 33 of 60

On this page

The platform offers a free test, and it would answer a smaller question than the one the CFO asked. The design is in holdouts, geo tests and power; the judgment is choosing which one.

The situation

A home-goods brand spends $200,000 a month on paid social. It sells $6,000,000 a month at 50% gross margin, 60% online and 40% through retail partners, reported weekly across 40 sales regions. The ad platform credits social with $900,000 of online sales a month and offers a free conversion lift test: a 5% person-level holdout, run and read by the platform, counting only the online orders it can match to its own users.

The CFO wants to know, by the end of next quarter, whether the channel pays back. Ads can be switched off by region. A region's six-week sales land within ±5% of a forecast built from its own history, at one standard deviation. Assume a two-sided 5% significance level and 80% power, so the smallest effect a test can see is (1.96 + 0.84) = 2.8 standard errors.

The question

Design the test that answers the CFO: the unit, the share held out, the length, the metric and the decision rule. Say what it costs.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The unit is the region. The platform's test cannot see the 40% of sales made in shops, and it is read by the party being graded; use it free as an online cross-check, not as the decision.

Break-even first: $200,000 of media at 50% margin needs $400,000 of incremental sales, 6.7% of $6,000,000. The platform's claim, $900,000, is 15%.

Switch social off in 10 of the 40 regions, one chosen at random from each of ten groups of four regions with similar history:

standard error   5% × √(1/10 + 1/30) = 5% × 0.365 ≈ 1.8%
smallest effect  2.8 × 1.8% ≈ 5.1% of sales, below the 6.7% break-even

Run six weeks dark, then count two more weeks after the ads return, because an ad's effect carries over. The metric is total sales per region, online plus retail, against each region's forecast, from our own systems.

The rule, written before launch: an effect of 6.7% or more keeps the budget; below it, cut spend in steps until the next dollar breaks even; an interval wholly under 6.7% halves it at once.

Cost: the dark regions buy no media, 25% × $200,000 × 1.4 months ≈ $70,000 saved. If the claim is right, they forgo 25% × 15% × $6,000,000 × 1.4 ≈ $315,000 of sales, $158,000 of margin, so the test costs about $88,000 net. At break-even it costs nothing, and if social does nothing it saves $70,000.

What would change the design: a 10% forecast error doubles the smallest visible effect to about 10%, above break-even, and even 20 dark regions see only 8.9%. The fix then is a tighter forecast, not more dark regions.

Rubric

Chooses regions as the unit and says why the platform's person-level test cannot settle the question
Derives the break-even effect, 6.7% of sales, from spend and margin
Sizes the dark share with the standard-error arithmetic and shows the test can see an effect below break-even
Sets a length that includes a post-period for carryover
Writes the decision rule, with its threshold, before the test runs
Costs the test both ways, counting the media the dark regions do not buy

Where answers go wrong

"Take the platform's 5% lift test; it is free." It never sees retail, so a channel that sells in shops looks worse than it is, and a 5% holdout is the slowest share to read. The seller also marks its own work.

"Switch social off everywhere for a month and compare with the month before." Season, promotions and the retail calendar all changed too; with no regions left running there is nothing to compare against.

"Hold out half the regions to be safe." Twenty dark regions see 4.4% instead of 5.1%, a small gain in precision for double the forgone margin when the channel works.