Evals, observability and guardrails · Evals

How many items before you believe a 2-point change?

Size an extraction feature's golden set to see a 2-point drop overall and a 5-point drop per slice, and cost the labelling and each run.

estimatemediumExercise · 3 min · 49 of 77

On this page

A team wants its eval to catch a 2-point drop before a model update ships. The set they have is too small to see one, and the estimate says how much bigger it must be and what that costs.

The situation

An invoice-extraction feature reads a supplier invoice and returns supplier, dates, currency, line items and total; an item passes only if every field matches its label. It passes 95% of a 150-item golden set. A model update is due, and the team wants a gate that catches a 2-point drop overall and a 5-point drop in each of four supplier-format slices, one of which — scanned PDFs — is 7% of traffic.

Labelling an invoice takes about 3 minutes, and a second look at a disputed one about 5. Assume the course constants: P_in = $3 and P_out = $15 per million tokens, decoding at 50 tokens/s, about a second to the first token. Each item is about 3,000 tokens in and 400 out, and the harness runs 10 items in flight. Treat a change as believable when it is at least two standard errors. The lesson this practises is regression gates.

The question

Estimate how many items the set needs, what growing it costs in labelling time, and what one run costs in money and minutes, then decide whether it is worth doing.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Overall: seeing 2 points at two standard errors needs a standard error of 1 point. √(0.95 × 0.05 ÷ n) = 0.01 gives n = 0.0475 ÷ 0.0001 = 475, call it 480. The pass rate matters: at 90% the same change needs 900, because p × (1 − p) nearly doubles.

Slices: 5 points at two standard errors needs 2.5 points, so n = 0.0475 ÷ 0.025² ≈ 76, call it 80 a slice, 320 for four. That fits inside 480 if the set is stratified. A random 480 would hold about 34 scanned PDFs, so oversample them to 80 and weight the overall by traffic share.

Labelling: 480 − 150 = 330 new items × 3 minutes = 990 minutes, 16.5 hours, plus a second look at the 15% that disagree, 50 × 5 minutes, about 4 hours. Roughly 20 hours: two and a half days of one person.

One run: input 480 × 3,000 = 1.44M tokens × $3/M ≈ $4.30; output 480 × 400 = 192k × $15/M ≈ $2.90; about $7.20. Time: 1 s plus 400 ÷ 50 = 8 s of decode is about 9 s an item, and 480 × 9 ÷ 10 ≈ 430 s, about 7 minutes. Three repeats of two versions: $43 and 45 minutes.

Worth it. The labelling is the cost and is paid once; runs are cheap enough for every change. The paired diff does better than the formula: if 3% of items flip between versions, its standard error is √(0.03 ÷ 480) ≈ 0.8 points.

What would change the answer: if only a 4-point drop overall matters, n = 0.0475 ÷ 0.0004 ≈ 120, and the four 80-item slices, 320, become what sizes the set.

Rubric

derives n from the standard error with the arithmetic shown, and says how the pass rate changes it
sizes the slices separately, stratifies the rare one and weights the overall
converts the new items to labelling hours, second look included
costs one run with input and output separately, and its duration from the decode speed
decides whether it is worth it and names the requirement that would change n

Where answers go wrong

"A thousand items, to be safe." No arithmetic. It buys precision nobody asked for at more than 40 hours of labelling, and if sampled at random it still leaves scanned PDFs short of a usable slice.

"150 is fine: losing 3 items is a 2-point drop, and we would see it." This confuses the size of a change with the power to see one. At 150 items the standard error is √(0.0475 ÷ 150) ≈ 1.8 points, so a 2-point drop is about one standard error, no different from a rerun.

"Runs are expensive, so run the eval monthly." A run is about $7. Labelling is the cost, and a gate that runs monthly misses the model update it was built to catch.