Evals, observability and guardrails · Evals

Ship or hold: 91% to 89%

Decide whether a cheaper triage prompt that scored 178 against 182 of 200 ships tomorrow, from the standard error, the paired diff and the cost of evidence.

casehardExercise · 3 min · 48 of 77

On this page

A rewrite that cuts the bill by 30% scored two points lower the day before launch. The decision turns on what two points on 200 items are worth, and on which tickets moved.

The situation

A support-triage prompt is rewritten to cut its system prompt from 6k to 2k tokens, saving about 30% of the feature's monthly spend: roughly $1,800 of a $6,000 month, $60 a day. Most of what the rewrite removed was worked examples, including the two that showed billing-versus-account tickets.

The golden set has 200 labelled tickets, sliced as billing 50, bug 45, how-to 45, account 35 and other 25. The old prompt scores 182/200 (91%); the new one 178/200 (89%). The harness stores a pass or fail per ticket for both runs, so the item-level results exist, but nobody has opened them. Each eval run makes its 200 calls one at a time; it costs $4 and takes 6 minutes. Launch is tomorrow, and the product manager has asked for a yes or no by the end of today.

The method this practises is in regression gates.

The question

Ship, hold, or something else — and what evidence would change your mind?

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

A pass rate of 0.9 on 200 items has a standard error of √(0.9 × 0.1 ÷ 200) ≈ 2.1 points, so 91 and 89 are one standard error apart: the set cannot tell these prompts apart. The totals are the wrong thing to read.

Look at the paired diff, which the harness already holds: which tickets flipped, and which way. A net −4 means failures minus gains is 4 — seven against three, say. Seven new failures all in the billing slice is a regression: billing drops 14 points, in the slice whose examples the rewrite cut. Seven scattered failures against three gains is noise: −4 on ten flips is about 1.3 standard deviations, with √10 ≈ 3.2.

Run each prompt three more times — 6 × $4 = $24, about 40 minutes — to measure run-to-run noise. The evidence costs less than half a day's saving and delays nothing past tomorrow.

Decision: hold for the 40 minutes, then ship if the flips are balanced across slices and the losses do not repeat on the same tickets. If they cluster in billing, hold the launch, restore the two billing examples — a few hundred of the 4k tokens saved — and re-run.

Then change the eval itself. Seeing a 2-point change at two standard errors needs √(0.9 × 0.1 ÷ n) ≈ 1 point, so n ≈ 900. Adding 100 billing tickets this week is a start, not the fix: at 300 the set still cannot see 2 points.

Rubric

computes what a 200-item set can and cannot detect
asks for the paired diff and the per-slice view
prices the extra evidence in the same units as the saving
states the decision and the evidence that would reverse it
turns the episode into a change to the eval itself

Where answers go wrong

"89 is below 91, so hold." It treats a noisy measurement as exact, and would also have blocked a change that made the feature 2 points better. The interviewer is scoring whether the candidate knows what the number is worth.

"Two points is inside the noise, so ship." Right about the totals, wrong to stop there. The same totals arise from seven losses packed into billing — a real regression in the slice the rewrite touched — and this answer never opened the flips that would show it.

"Re-run the new prompt once and ship if it gets back to 182." One more run is one more noisy draw, and drawing until a run looks good is how a regression ships. Repeats measure the spread of both prompts; they are not a second chance for one of them.