Turn a baseline rate and the lift worth shipping for into users per arm, and read the calendar off the result.
codemediumExercise · 4 min · 26 of 59
On this page
Every experiment the team plans starts with the same sum, and twice last quarter the sum was done wrong by a factor of a hundred. The function below is the sum done once.
The analytics team runs experiments for the drafting assistant on three kinds of metric: the help-centre self-serve rate for one tenant (5.0% baseline, 2,000 visitors a day), draft acceptance across all agents (40% baseline), and trial-to-paid conversion on the self-serve tier (6% baseline). Each test is sized in a planning meeting by whoever has the formula in a notebook, and twice last quarter the minimum detectable effect was entered as an absolute gap rather than a fraction of the baseline, so a test that needed 30,000 users per arm was planned with 76, declared "no significant difference" after a day, and the feature was dropped.
Assume the standard settings of 5% significance and 80% power, and the rule of thumb from reading an experiment: users per arm n ≈ 16 × σ² ÷ δ², with σ² = p × (1 − p) for a rate metric and δ = p × lift the absolute gap to detect. A baseline is measured to about two significant figures, so the answer carries two.
Write the function and use it to say how long the help-centre test runs at 2,000 visitors a day.
def solve(baseline_rate, relative_lift):
"""Users per arm needed to detect a relative lift on a rate metric.
baseline_rate: the control's rate as a fraction, 0 < p < 1 (0.05 for 5%).
relative_lift: the minimum detectable effect as a fraction of the baseline (0.10 for +10%).
Returns an int: the rule-of-thumb count per arm, rounded to two significant figures.
"""
The body is three lines and a rounding. Variance is p × (1 − p): 0.05 × 0.95 = 0.0475. The gap is p × lift, not the lift: 0.05 × 0.10 = 0.005, and this is the line the meeting got wrong, because entering 0.10 gives 16 × 0.0475 ÷ 0.01 = 76 instead of 30,400. Then 16 × variance ÷ gap² = 16 × 0.0475 ÷ 0.000025 = 30,400, where 16 is the 5%/80% settings folded in: 2 × (1.96 + 0.84)² ≈ 15.7.
The rounding is a deliberate choice. 30,400 implies the baseline is known to three figures, and it is not; last month it was 5.2%. Rounding to two significant figures gives 30,000, which is also what the lesson's calendar arithmetic uses: two arms is 60,000 visitors, and at 2,000 a day the help-centre test runs for 30 days. The rounding helper finds the number of digits, divides by the matching power of ten, rounds half up and multiplies back; under 100 it rounds to a whole user.
The function never accepts a baseline outside (0, 1) or a non-positive lift; it raises, because a sample size of zero or a negative one would be read as "no test needed". It returns an integer. And it does not try to be the exact two-proportion formula: the rule and the formula differ by about 3% here, which is less than the drift in the baseline, and the rule is the one a PM can do aloud.
What would change the answer: a metric that is not a rate. Time to resolution has its own variance, measured from the data, and p × (1 − p) does not apply; the function's docstring says "rate metric" for that reason.
"delta = relative_lift" — the gap is entered as 0.10 instead of 0.005, the result is 76 per arm, and the test ends in a day with nothing learned. This is the mistake the situation describes, and it passes any test that only checks the function runs.
"Return the exact figure, 30400" — the number is not wrong, but it reports a precision the baseline does not have, and a team that sees 30,400 plans to the day rather than to the week. The rounding is the product decision in this exercise.
"Divide the per-arm number by 2,000 and get 15 days" — forgets the control arm. The test needs 30,000 in each of two arms, so the calendar is 30 days, and a plan that books 15 reads its result at half the sample.
Python 3 · standard library27 lines · 6 of 6 tests passing
"""Sample size per arm — Kohavi, Tang and Xu's rule of thumb, n ≈ 16 σ² ÷ δ², at 5% significance and 80% power."""
import math
def two_figures(n):
"""Round to two significant figures, half up: 30,400 -> 30,000; 25,600 -> 26,000; 96 -> 96.
A baseline measured last week carries two figures, so the sample size carries no more."""
if n < 100:
return int(math.floor(n + 0.5))
unit = 10 ** (len(str(int(n))) - 2) # 30,400 has 5 digits, so the unit is 1,000
return int(math.floor(n / unit + 0.5)) * unit
def solve(baseline_rate, relative_lift):
"""Users per arm needed to detect a relative lift on a rate metric.
baseline_rate: the control's rate as a fraction, 0 < p < 1 (0.05 for 5%).
relative_lift: the minimum detectable effect as a fraction of the baseline (0.10 for +10%).
Returns an int: the rule-of-thumb count per arm, rounded to two significant figures.
"""
p = baseline_rate
if not 0 < p < 1 or relative_lift <= 0:
raise ValueError("baseline_rate must be in (0, 1) and relative_lift positive")
variance = p * (1 - p) # σ² of a yes/no metric
delta = p * relative_lift # the absolute gap to detect — never the lift itself
n = 16 * variance / (delta * delta) # the 16 folds in z for 5% and 80%
return two_figures(n)
The cases that ran
TESTS = [
((0.05, 0.10), 30_000), # the help-centre test: 30,400 by the rule, 30,000 to two figures
((0.20, 0.05), 26_000), # a higher baseline, a smaller lift
((0.50, 0.10), 1_600), # a coin-flip baseline needs few users
((0.02, 0.25), 13_000), # a rare event needs a large lift to be testable
((0.10, 0.02), 360_000), # a 2% lift is a six-figure test
((0.40, 0.50), 96), # a huge lift on a high baseline: under a hundred
]
