Turn a baseline, the lift that matters and the share held out into the total audience a lift test needs, and read the weeks off the result.
codemediumExercise · 4 min · 34 of 60
On this page
A lift test sized as if the split were 50/50 and run with a 5% holdout is a test that was never going to see anything. The function below sizes it for the split it will actually run with, as holdouts, geo tests and power works by hand.
The growth team plans conversion lift tests on three paid channels a quarter. Its planning sheet sizes every test per arm, assuming equal groups, but the team asks the ad platforms for a 5% or 10% holdout to protect sales. Last quarter a test sized at 81,000 per arm, the lesson's 2.0% to 2.2% example, ran on 162,000 people with 10% held out. It could only have seen a lift of about 16%, read "not significant", and the channel was cut.
Assume a two-sided 5% significance level and 80% power, with z values from the standard normal distribution (1.96 and 0.84), and the lesson's formula for unequal groups, where p₁ is the held-out rate, p₂ = p₁ × (1 + lift) and h is the share held out:
total people = (z_α/2 + z_β)² × [p₁(1 − p₁) ÷ h + p₂(1 − p₂) ÷ (1 − h)] ÷ (p₂ − p₁)²
Round the total up to the next 1,000, never down: a test smaller than its size has less power than promised. The paid-social program reaches 2 million people a month, converts 0.10% of them without ads, and the lift that changes its budget is 15%.
Write the function, then use it to decide which holdout share answers the paid-social question inside one quarter.
def solve(baseline, relative_lift, holdout_share=0.5, alpha=0.05, power=0.80):
"""Total people (held out plus exposed) a conversion lift test needs.
baseline: the held-out group's conversion rate, 0 < p < 1 (0.02 for 2%).
relative_lift: the lift to detect, as a fraction of the baseline (0.10 for +10%).
holdout_share: the share of the audience that sees no ads, 0 < h < 1.
alpha, power: two-sided significance level and power.
Returns an int: the total, rounded UP to the next 1,000.
"""
The z values come from the arguments, not from memory: NormalDist().inv_cdf(1 − α/2) and inv_cdf(power), so a 90% power request costs what it should. The exposed rate is the baseline times one plus the lift, and the gap is their difference, never the lift itself. Each group's variance, p(1 − p), is divided by its share of the audience. That one line is what the planning sheet lacked: at h = 0.5 it gives exactly twice the per-arm formula, 2 × 80,678.6 ≈ 161,357, rounded up to 162,000; at h = 0.05 the held-out term is ten times larger and the total is 814,000.
The rounding is ceil(total ÷ 1000) × 1000. Rounding to the nearest would turn 161,357 into 161,000, a test slightly smaller than its stated power; at 90% power the total is 216,012, which must become 217,000, not 216,000. The function raises on a rate outside (0, 1), a non-positive lift or a share outside (0, 1), because a size computed from impossible inputs would be read as a plan. The tests pin 162,000, 814,000 and the floor: a 30% baseline lifted by half needs 319 people, rounded up to 1,000.
Reading the results for paid social at 2 million people a month: 7,392,000 at a 5% holdout is 3.7 months, about 16 weeks, past the quarter. At 20%, 2,244,000 is about 5 weeks. Choose 20%: it holds out 449,000 people against 370,000 at 5%, a small extra cost for an answer eleven weeks sooner.
What would change the answer: a dollar outcome such as revenue per person instead of a yes/no conversion. Its variance has to be measured, and Lewis and Rao found it is often a hundred times the squared mean.
"Compute 81,000 per arm, double it, and let the platform hold out 10%." This is last quarter's test. The per-arm formula assumes equal groups; with 16,200 people held out the test could see only a 16% lift.
"Return round(total, -3)." 161,357 becomes 161,000 and the test runs under its stated power, and the function returns a float where a count of people is an integer.
"Use the rule of thumb 16 × p(1 − p) ÷ gap², doubled." It is fine for a 50/50 product test, as the ai-pm course's sample-size exercise uses it, but it hard-codes 80% power and equal groups, the two things this function exists to vary.
Python 3 · standard library37 lines · 7 of 7 tests passing
"""Sample size for a conversion lift test with an unequal split: total people, held out plus exposed."""
import math
from statistics import NormalDist
def round_up_to_thousand(n):
"""Round a sample size UP to the next 1,000: 161,357.3 -> 162,000; 319.2 -> 1,000.
Rounding down would run the test below the size its power was promised at, so never round down."""
return int(math.ceil(n / 1000)) * 1000
def solve(baseline, relative_lift, holdout_share=0.5, alpha=0.05, power=0.80):
"""Total people (held out plus exposed) a conversion lift test needs.
baseline: the held-out group's conversion rate, 0 < p < 1 (0.02 for 2%).
relative_lift: the lift to detect, as a fraction of the baseline (0.10 for +10%).
holdout_share: the share of the audience that sees no ads, 0 < h < 1.
alpha, power: two-sided significance level and power.
Returns an int: the total, rounded UP to the next 1,000.
"""
p1 = baseline
p2 = baseline * (1 + relative_lift) # the exposed group's rate if the lift is real
h = holdout_share
if not (0 < p1 < 1 and 0 < p2 < 1 and relative_lift > 0 and 0 < h < 1 and 0 < alpha < 1 and 0 < power < 1):
raise ValueError("rates must lie in (0, 1), the lift must be positive, and the share in (0, 1)")
z = NormalDist()
z_alpha = z.inv_cdf(1 - alpha / 2) # 1.96 for a two-sided 5% test
z_power = z.inv_cdf(power) # 0.84 for 80% power
gap = p2 - p1 # the absolute difference to detect, never the lift itself
# Each group's variance is divided by its share of the audience: a small
# holdout is measured on few people, so its term dominates the sum.
# At h = 0.5 this is exactly twice the familiar per-arm formula.
variance = p1 * (1 - p1) / h + p2 * (1 - p2) / (1 - h)
total = (z_alpha + z_power) ** 2 * variance / gap ** 2
return round_up_to_thousand(total)
The cases that ran
TESTS = [
# The lesson's worked test, 2.0% -> 2.2% at 50/50: 80,679 per arm, 161,357 in all.
((0.02, 0.10), 162_000),
# The same test with 5% held out: five times the people.
((0.02, 0.10, 0.05), 814_000),
# The $200k-a-month channel: 0.100% -> 0.115% with 5% held out, about 16 weeks at 2 million a month.
((0.001, 0.15, 0.05), 7_392_000),
# The same channel at 20% held out, about 5 weeks.
((0.001, 0.15, 0.20), 2_244_000),
# Sized to see the platform's whole claim, a 30% lift, at 5% held out.
((0.001, 0.30, 0.05), 1_862_000),
# 90% power instead of 80% on the lesson's 50/50 test: 216,012 people, so up to 217,000, not down to 216,000.
((0.02, 0.10, 0.5, 0.05, 0.90), 217_000),
# A large lift on a high baseline needs 319 people; the floor of the rounding is 1,000.
((0.30, 0.50), 1_000),
]
