Regression gates
Gate a prompt or model change on the paired diff, per-slice losses, repeats, cost per task and p95, instead of one pass rate that cannot see a 2-point move.
On this page
A regression gate is the rule that turns an eval run into ship, hold or block. The obvious rule — block if the pass rate fell — fires on noise about half the time and lets through a real regression concentrated in one slice. A pass rate on 200 items has a standard error near 2 points, so the gate has to look at which items moved, where they sit, and whether they move again on a repeat, and it has to read cost and latency from the same run.
What one total can and cannot say
The standard error of a pass rate p over n items is √(p × (1 − p) ÷ n). For the triage set at 91%, that is √(0.91 × 0.09 ÷ 200) ≈ 2.0 points. The 200 tickets are a sample of the traffic; a different 200 would have scored a couple of points either way. Two prompts scoring 182 and 179 are 1.5 points apart, inside one standard error of either.
Now the naive gate: block if the new total is below the old. When a change does nothing, the new total is as likely to land below as above, so the gate blocks about half of all harmless changes and the team learns to override it. Loosen it to "block on a drop of more than 4 points" and a real 3-point regression passes most of the time. No threshold on one total fixes both.
A second noise source sits on top. Boundary items flip between runs of the same prompt — the near-ties from sampling and determinism — so one prompt run three times may score 182, 180 and 183. A gate needs a handle on both kinds of noise.
The paired diff
Both versions ran on the same 200 items, so compare them item by item. Every item lands in one of four cells: passed both, failed both, passed before and failed after (a loss), or the reverse (a gain). Items that passed both or failed both say nothing about the change. Only the flips do.
A worked case. The provider updates the model behind the name; the prompt is unchanged; the nightly run on the triage set goes from 182 to 180.
If the change made no difference, each flip is as likely to go one way as the other, like a coin. With f flips, the net — gains minus losses — has a spread of about √f items. Here the net is −2 against √14 ≈ 3.7: half a standard deviation, nothing. Had the 14 flips been 12 losses and 2 gains, the totals would read 182 and 172 and the net of −10 would be 2.7 standard deviations: a regression. This is the sign test on the flips — McNemar's test, in a statistics text — and it is a few lines in the harness:
losses = [i for i in ids if before[i] and not after[i]]
gains = [i for i in ids if after[i] and not before[i]]
net, flips = len(gains) - len(losses), len(gains) + len(losses)
z = net / flips ** 0.5 if flips else 0.0 # |z| >= 2: a real move
Pairing is why this is sharper than comparing totals: the 186 items that did not flip cancel exactly, so the noise is the flips' noise, not the whole set's.
Slices: where the total hides a regression
Same run, by slice. Six of the eight losses are in account, a 35-item slice, with no gains there.
Account falls from 32 to 26, 91% to 74%. Six losses and no gains in one slice is a coin landing the same way six times, about 3% likely by chance, and account is where a misroute delays a customer trying to close their account. The total and the paired total both said noise. The slice said regression.
So the gate checks slices as well as totals, with a written rule for each check:
| Check | Fires when | Then |
|---|---|---|
| must-pass items | any failure among them (injection attempts, closures) | block |
| slice | a slice of 20 or more items loses 4 or more net | hold for review |
| paired total | net flips at 2 or more times √flips | hold for review |
| cost per task | more than 10% above the baseline | hold, unless the change is a declared trade |
| p95 latency | above the feature's budget | block |
The thresholds are settings, not laws: each is written down, applied to every run, and loose enough that a change which does nothing almost always passes.
Repeats measure the noise floor
When the decision is close, run each version three times. The repeats show the run-to-run spread — if the old prompt scores 182, 180 and 183, a 2-item gap between prompts is nothing — and they mark the unstable items, the ones that flip between runs of the same prompt. Exclude those from the paired diff or count them as half; a loss on an item that flips by itself is not the new version's doing.
Three repeats of two versions is six runs. For the triage set each run is 200 calls of about 750 tokens in and 5 out, 150k × P_in. At the course's latency figures (about 1 s to the first token and 50 tokens/s, from the four numbers) a call takes about 1.1 s, so with ten in flight a run is 20 rounds, about 20 seconds, and the six runs take about two minutes. The evidence is almost always cheaper than shipping the wrong prompt. The ship-or-hold exercise prices it against a saving.
Cost and p95 gates beside accuracy
A gate that watches only the pass rate waves through a change that wins 2 points by spending far more. Add "think through the ticket before you label it" to the triage prompt and output rises from 5 tokens to about 200. At 50 tokens per second that is 4 more seconds of decode, so p95 goes from under 2 seconds to about 6. Output tokens per call rise 40×: at 10,000 calls a day, 2M × P_out a day where it was 50k × P_out. The 2 points may be worth it. The gate's job is to make the trade explicit rather than discovered on the invoice.
Measure both in the eval run itself, from the usage counters and the wall clock per case — the numbers where the money goes and latency budgets and streaming build the cost sheet and the budget from. Gate on cost per task, not per call: a change that adds a turn to an agent loop leaves cost per call flat and raises cost per task by a third or more. Gate on a percentile, not the mean: 200 cases give a usable p95, about the tenth-slowest case, and the tail latency argument applies unchanged.
In an interview
What is tested is whether you know what a number is worth. Say: "on 200 items the standard error is about 2 points, so I gate on the paired diff and per-slice losses, not the total; I re-run three times when it is close; and cost per task and p95 are gates in the same run." Then do the arithmetic on the numbers you are given, out loud.
The mistake that loses points is reading 89 against 91 as a regression — or as no change — without saying how big a move 200 items can see. The second is a gate with no cost or latency check, which ships a prompt that is 2 points better and four seconds slower.
Check yourself
1. A change produces 12 flips: 9 losses and 3 gains, spread across four slices. Ship?
Net −6 against √12 ≈ 3.5 is about 1.7 standard deviations, close to the line, and no slice fires. Run both versions twice more. If the losses repeat on the same items, it is a small real regression and the question becomes whether the change bought something worth 3 points; if the losses move around between runs, they are unstable items and the change ships.
2. A revision gains 3 points and output per call goes from 5 tokens to 150. The feature makes 10,000 calls a day against a 3-second p95 budget. What does the gate say?
Decode adds 145 ÷ 50 ≈ 3 seconds, which breaks a 3-second budget on its own: block on p95. Output goes from 50k to 1.5M tokens a day, 1.5M ×
P_out. If the 3 points are wanted, find them another way — a tie-break rule costs a few input tokens — or move the reasoning off the user's path.
3. How many repeats would you budget for a launch decision, and what happens to an item that flips between two runs of the same prompt?
Three runs of each version: enough to see the spread and mark unstable items, for six runs of tokens and well under an hour. An item that flips within one prompt is a boundary case: exclude it from the paired diff, move it to the known-ambiguous slice, and give it a tie-break rule or a repeat-test agreement number instead of a pass or fail.