What AI changed: content, search, research · Personalisation and testing

Design a creative testing plan for 50 variants

Design a screen-then-confirm plan for 50 generated ad variants on one account's traffic, with metrics, sample sizes and the rule for shipping.

designmediumExercise · 3 min · 43 of 60

On this page

Fifty generated ad variants and one account's traffic. The plan has to say what is tested on what, for how long, and what is allowed to ship.

The situation

A generation tool produced 50 paid-social creative variants in an afternoon for a company larger than module 2's invoicing app. Its account spends $100,000 a month at the course's rates, a $20 CPM and a 1% CTR: 5,000,000 impressions and 50,000 clicks a month at $2 a click. 3% of clicks sign up, 1,500 sign-ups a month. The current ad is the control. The creative lead's plan: split traffic evenly across the 50 variants and the control for a month, about 980 clicks each, and roll out every variant that beats control at p under 0.05. Use the course's test settings, two-sided α = 0.05 and 80% power, and assume: detecting a 15% relative lift on a 3% sign-up rate takes about 24,000 clicks an arm, or about 29,000 an arm with α split between two comparisons. On a 1% CTR, about 49,000 impressions an arm detect a lift of roughly 20%. The team would ship a variant worth a 15% lift or more.

The question

Design the testing plan for the 50 variants: stages, metric, traffic and duration for each, and the rule for what ships.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The lead's plan cannot work. At 980 clicks an arm, about 29 sign-ups each, a variant must beat control by roughly 50% to read as significant, and nothing real is that large. With 50 comparisons at α = 0.05, 2.5 false winners are expected when nothing works, about half above control and shipped, and any real winner arrives with its lift inflated because it was picked for being highest.

Two stages, with different jobs.

Screen, two weeks. Rotate all 50 and the control on impressions, 2,500,000 ÷ 51 ≈ 49,000 each, reading CTR. It detects only lifts near 20%, which is fine: its one output is a shortlist of the top 2. Nothing ships from it and no lift from it is reported.

Confirm, seven to eight weeks. The 2 finalists against control on fresh traffic, measured on sign-ups, the metric that pays, because a creative can win clicks and lose sign-ups. At 29,000 clicks an arm with α split two ways, that is 87,000 clicks, 1.7 months at 50,000 a month. The horizon is fixed in advance and read once; if the team must look daily, use an always-valid sequential method instead.

Ship rule. A finalist ships only if it beats control on sign-ups in the confirmation, and the forecast uses the confirmation lift.

Standing holdout. From day one, keep 5% of impressions on the old control permanently, so the program's claimed lifts can be checked against what sign-ups did. It stretches the confirmation by a few days.

What would change the plan: a finalist whose CTR lead vanishes on sign-ups. Then CTR is the wrong screen for this account; next round, screen fewer variants on sign-ups directly.

Rubric

shows why the even split fails, with the per-arm sample and either the detectable lift or the false-winner count
separates a screen from a confirmation, with metric, traffic and duration for each
confirms on sign-ups rather than CTR, and says why
adjusts the confirmation for the number of comparisons
fixes the horizon or names a sequential method, and reports the confirmation lift rather than the screen's
adds a standing holdout, or states the result that would change the plan

Where answers go wrong

"Keep the even split but run it for three months." About 2,900 clicks an arm still needs close to a 30% lift to read as significant, and the 2.5 expected false winners do not shrink with time; the plan buys a slower version of the same error.

"Apply Bonferroni across all 50 and ship whatever survives." At α = 0.001 and 980 clicks an arm, a variant needs roughly an 85% lift to pass, so a real 15% winner is all but invisible. The correction makes the test honest and blind; the lesson uses it on a two- or three-way confirmation instead.

"Ship the top CTR variant after the two-week screen." That is the winner's curse on a proxy metric: a lift selected for being highest, on clicks rather than sign-ups, booked as if it were measured.