Evals with the customer's definition of good · Grading

The 94% that meant nothing

Find what a colleague's 94% actually measured, decide what to run before a Monday prompt switch, and correct the number with the sponsor the same day.

critiquehardExercise · 4 min · 44 of 64

On this page

A number reached the sponsor before it reached you, and it is higher than anything the pilot has measured. The work is to say precisely what it measured, and to correct it without making the colleague who sent it the story.

The situation

Week 5 of the claims-triage pilot: the shadow run at 500 claims a day, which the per-segment gate reads at the end of week 6. A solutions engineer from your own company, helping for a fortnight, sent the VP of Claims a slide on Friday: prompt v6 scores 94% on 500 claims, "better than your handlers agree with each other (87%)", and he proposes switching the shadow run to v6 on Monday. You check how it was made. The 500 are last week's claims from the export; 49 are commercial, against 20% of live volume. The label is the escalation decision the claims system recorded at triage, set by whichever of the 28 handlers took the claim. While writing v6 he read 60 of the 500 that v5 got wrong and added four of them as worked examples. Each v6 rationale was scored by a judge from the same model family as the pilot, asked whether it "supports" the recorded decision on a 1-to-5 scale, with 4 or 5 counted as agreement. On the adjudicated 200, v5 reads 185, 92.5%, with one missed escalation.

The question

Find the defects in the 94% as evidence for switching to v6, and decide what you run and what you tell the VP before Monday.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

The 94% measures something other than the definition of done, four ways at once.

The label is the floor's unreviewed decision, the process the pilot exists to beat. On unadjudicated labels reviewer agreement is the ceiling, and 87% is the senior handlers'; scoring above it is the tell, as an eval is not a benchmark warns. v6 has learned to reproduce the floor, mistakes included.

The set graded the prompt it trained. Sixty of the 500 were studied and four copied into v6, so part of the score is v6 recognising its own examples.

The population is wrong. Commercial is 49 of 500, 10% against 20% live, so the hardest segment counts half, and one week is one weather pattern. No per-segment figure was reported, and no missed-escalation count.

The grader adds bias where none was needed. The decision is a label, so exact match grades it for free; a same-family judge on a lenient 1-to-5 "supports" scale risks self-preference, uncalibrated against any person.

Before Monday, run v6 on the adjudicated 200 with exact match: about 1r and an hour. Report agreement, per segment, missed escalations and precision on escalate beside v5's. Switch only if v6 reaches 190 or more, the agreement line and five claims ahead, with no more missed escalations.

To the VP on Friday, with the solutions engineer beside you: "The 94% was measured on a different test from the one in your definition of done. You'll have v6's number on your 200 on Monday morning, and the switch is decided on that." Correct it the same day: a number left in the VP's deck becomes the target.

Rubric

identifies the label as the floor's unreviewed decision and says why beating 87% on it is a warning, not a result
identifies the leakage from the 60 studied cases and the four copied examples
quantifies the segment shortfall, 49 of 500 against 20%, and notes the missing per-segment and missed-escalation figures
says why a same-family judge on a 1-to-5 scale was the wrong grader for a label
names the re-run on the adjudicated 200 with exact match, its cost, and the threshold for switching
corrects the number with the sponsor the same day, without making the colleague the story

Where answers go wrong

"94% is above the 92.5% we have; switch, and confirm it at the week-6 gate." Loses because the two numbers are on different sets, labels and graders, so comparing them compares nothing, and the switch puts an unmeasured prompt into the shadow run the week-6 gate is read on, a week before handlers see it in walk.

"Ignore the 94% and say nothing until v6 is re-measured." Loses because the VP has the slide; silence lets 94% become the expectation, and the real number, when it arrives, reads as a drop.

"The judge is the problem; re-score the 500 with exact match." Loses because exact match against the floor's decisions still measures fidelity to the floor, and the set still contains v6's own examples; it fixes the least important of the four defects.