When an A/B test is the wrong instrument
Recognise the four situations where an A/B test cannot answer the question — too few users, network effects, slow loops, a strategy — and pick the holdout, switchback, pre/post or interview that can.
On this page
An A/B test is the right instrument when there are enough independent users, the effect shows up inside the test window, and the thing being tested is a variant of something that exists. Each of those fails often in an enterprise AI product, and when one fails the test does not return a wrong answer so much as no answer, dressed as one. The skill is recognising which condition has failed and reaching for the instrument built for it.
Not enough users
The experiment lesson sized a 10% lift on a 5% baseline at about 30,000 per arm. A tenant with 1,000 help-centre visitors a week reaches 60,000 in 60 weeks. Run it for eight weeks instead and there are 4,000 per arm, which can detect only a lift of about δ = √(16 × 0.0475 ÷ 4,000) ≈ 0.014, 28% relative. If the feature is worth shipping at a 10% lift, that test cannot see the thing it is looking for, and "no significant difference" after eight weeks means nothing.
The rule of thumb: under about 1,000 users a week, an A/B test on a rate metric detects only large effects, and a feature whose honest hope is a 10% lift should not be put in front of one. The enterprise version is worse: the 12,000 agents sit in 40 tenants, and for anything that varies by tenant — a prompt tuned to their macros, a retrieval index of their articles — the unit is the tenant, so the sample is 40, and there is no test.
Users who affect each other
An A/B test assumes one user's treatment does not change another user's behaviour. Agents on the same tenant share a queue, a set of macros and a Slack channel. Give half of them drafts that are 30 seconds faster, and the other half pick up the slack on the queue, see the drafts forwarded, or copy the phrasing. The control is contaminated; the measured lift is smaller than the real one, or larger, and there is no way to say which.
Where users interact, randomise at the level that contains the interaction. For the queue, that is the tenant, which brings back the 40-unit problem. For a marketplace-shaped effect — the assistant's drafts change how fast the queue drains, which changes which tickets the next agent gets — the instrument is a switchback: the whole tenant alternates between control and variant by day or by hour, and the comparison is across periods rather than across people. Switchbacks need the effect to wash out between periods, so a draft feature that changes the queue for an hour needs a period longer than an hour.
Loops longer than the test
The metric that matters for the drafting assistant is the North Star, and one of its inputs is the 7-day no-reopen rate; another thing a tenant cares about is week-4 retention of agents on the tool. A two-week test cannot read a week-4 number, and a test that reads draft acceptance at two weeks and calls it done has measured the input and assumed the output.
Loops also run the other way. If accepted drafts feed the next prompt's examples, the variant improves itself during the test and the control does not, and the gap at day 30 is partly the feedback loop rather than the change. The slow loop is not a reason to skip measurement; it is a reason for a holdout: keep 5% of tenants or agents on the control for a quarter, read the North Star for both groups at the end, and accept that the holdout costs 5% of the lift for three months. That cost is the price of knowing whether the quarter's work moved the output rather than an input.
A strategy, not a variant
"Should the product add an assistant for the end customer?" is not an A/B test, because there is no variant: the test would compare a prototype built in a week against nothing and read a number that measures the prototype's polish. A test needs two versions of the same thing. For a strategy question the instruments are the ones from the discovery module — interviews, a one-day prototype in front of five customers — and the decision is a bet with its reasoning written down, not a p-value. The mistake is running the test anyway and letting a 4% lift on a prototype decide a year of roadmap.
Pre/post with a control metric
When there are too few units to randomise and the change goes to everyone, the fallback is before-and-after, which is unreliable on its own because everything else changed too: ticket volume, the season, the tenant's own hiring. The repair is a control metric: a number the change should not affect, measured over the same period. A tone fix to the drafting prompt was switched on for one tenant on 1 September; draft acceptance for that tenant went from 40% to 46%. The control metric is the same tenant's ticket volume, or acceptance at a comparable tenant that did not get the change. If the comparable tenant also went from 40% to 45%, five of the six points are the season and the fix earned one; if it stayed at 40%, the 6 points are evidence. The structure is a difference in differences, and the honest version names the control metric before the change ships, for the same reason the experiment lesson pre-registers the ship rule.
This is also the instrument for a launch ramp read at the tenant level, where the launch lesson compares the 5% stage against the tenants not yet in it.
Interview instead
Some questions are not about how much but about why, and no instrument that counts can answer why. "Agents accept 18% of long drafts" is a finding from the funnel; why they reject the other 82% is a question for five agents and a screen recording, and the answer arrives in an afternoon. The pattern: an instrument that counts tells you where; an interview tells you why; the next test checks the why. A team that only tests ships the hypothesis it happened to have; a team that only interviews never finds out how big anything is.
Choosing
Ask the four questions in order: is it a variant of something that exists; are there enough independent users for the lift worth shipping; do users affect each other; does the effect outrun the test window. The first failure picks the instrument. Most AI features in an enterprise product fail the second question, and the honest answer is a pre/post with a control metric and an interview, which is less satisfying than a p-value and considerably more accurate than a p-value from a test that could not see the effect.
In an interview
The execution question arrives as "how would you measure whether this worked?" and the trap is answering "A/B test" before asking how many users there are. Ask, size the test aloud, and when the arithmetic says sixty weeks, say which instrument replaces it and what it costs: "a 5% holdout for a quarter, read on the North Star, which forgoes 5% of the lift for three months". Naming the cost is what shows the instrument was chosen rather than recited.
The mistake that loses points is proposing an A/B test for a question it cannot answer — forty tenants, a strategy, a week-4 metric in a two-week window — and then reading "not significant" as "no effect". The second mistake is the opposite: refusing to measure at all because a test is impossible, when a holdout or a control metric was available.
Check yourself
1. A tenant with 800 help-centre visitors a week wants to test the cited-answer variant. The lift worth shipping for is 10% on a 5% baseline. What do you run?
Not an A/B test: 30,000 per arm at 800 a week is 75 weeks. Run a pre/post: switch the variant on for the tenant, pre-register ticket volume and a comparable tenant's self-serve rate as control metrics, read after four weeks. If both tenants moved together, it was the season. The evidence that would bring back the A/B test: pooling across tenants with the same help-centre template until the weekly volume passes about 5,000.
2. The assistant's drafts speed up one tenant's queue, and the team proposes randomising agents within the tenant. Do you agree?
No; faster drafts for half the agents change which tickets reach the other half, so the control is contaminated. Propose a switchback by day — whole tenant on the variant on alternate days for four weeks — and check the queue returns to baseline overnight so a day is a clean period. If it does not wash out overnight, lengthen the period to a week.
3. Leadership wants a test to decide whether to build the end-customer assistant. What do you propose instead, and how long does it take?
A one-day prototype in front of five customers and ten interviews over two weeks, with the decision written as a bet: what we believe, what we saw, what would reverse it. A test would compare a rough prototype against nothing and measure polish. The evidence that would reverse the bet is three of five customers not using the prototype a second time.