An eval is not a benchmark
Tell a leaderboard score from the customer's eval, set the ceiling from their reviewers' agreement, and know when a reading above it is noise.
On this page
A public benchmark scores a model on someone else's task, with someone else's labels and someone else's metric. The customer's eval scores the deployment on their claims, labelled by their reviewers, against the line they signed in the definition of done. Only the second number can be argued at renewal, and it has a ceiling: how often the customer's own reviewers agree with each other.
What a leaderboard score measures
Holistic Evaluation of Language Models (Liang et al.) makes the point in its structure: a score is a model measured on a scenario with a metric, and it means nothing outside that triple. Change the scenario or the metric and the number does not transfer.
The mistake arrives in week 2, usually from the sponsor: a slide shows another model three points ahead of the pilot's on a public benchmark, and should we switch? Hold the leaderboard against the triage pilot and four things differ:
- The inputs. The benchmark's questions are clean text. The pilot reads a 3-page form, a scanned adjuster note, and a 340-page policy set whose governing clause is often in an endorsement.
- The label. The benchmark's answer key was written by its authors.
escalateat this insurer means a claim routed to fraud review, such as a second claim on the policy within a year, or to a handler because the wording does not settle it, or any claim above the high-value band — rules nobody outside the building has seen. - The metric. The benchmark reports accuracy. The definition of done counts agreement on the 200, missed escalations, precision on
escalateand coverage; a model better on average and worse on escalations loses the pilot. - The population. The benchmark's mix is fixed. The floor's is motor 55%, property 25%, commercial 20%, and it moves with the weather.
A fifth difference favours the customer's set. Public test sets live on the open web, and a model trained on a crawl of it may have seen the questions, so a high score can be memory. The customer's 200 claims have never left their tenant. Where the role comes from makes the same point in product terms: the benchmark measures the inner box, and the pilot is sold on everything around it.
Checking the competitor's claim is cheap. A claim costs about 0.005r through the pipeline, so running a second model over the 200 costs 200 × 0.005r ≈ 1r and an afternoon. The answer to a leaderboard is not an argument about it; it is a run.
The gold label is the customer's
Gold labels come from the customer's senior handlers, under the QA lead who signed the scoping doc as owner of the set. Three other sources are tempting, and each fails the renewal conversation.
You, labelling. You grade your own work against your own reading of their policy, and the first time the head of underwriting disputes one of your labels, every number computed on the set is yours to defend.
The model, labelling, with a person approving. A reviewer approving a suggested label agrees with it far more often than one labelling from scratch, so the gold set inherits the model's errors and the eval measures the model against itself.
The claims system's historical escalation field. It is free and covers 78,000 claims, but it is the decision recorded by whichever of the 28 handlers took the claim, unreviewed: the current process, mistakes included, which the pilot exists to beat. A model that matches it at 94% has learned to reproduce the floor.
The customer's labels cost reviewer time, and the cost belongs in their units: 200 claims × 2 handlers × 11 minutes ≈ 73 reviewer-hours, the figure saying no uses when a second eval is requested in week 3.
The reviewers' agreement is the ceiling
Two senior handlers labelled the same 200 claims escalate or approve and agreed on 174: 87%. On 26 claims, two people with years on the desk read the same file and decided differently.
Suppose the gold set is handler A's labels alone. Handler B, as senior as A, scores 87% against it. A pilot scoring 94% against A is not a better handler than B. The seven points above B are agreement with A in particular: A's reading of the high-value band, A's habits on endorsements, and perhaps A's own cases, if the prompt's worked examples were taken from A's labels. The eval cannot tell agreement with A's habits from being right. Hence the working rule: on a gold set that has not been adjudicated, raw two-reviewer agreement is the practical ceiling, and a reading above it is a defect to investigate, not a result to report.
The ceiling has a band around it. On 200 cases one case is half a point, and the sampling noise on an agreement near 90% is about two points: √(0.9 × 0.1 ÷ 200) ≈ 0.021, about four claims. Two prompts run on the same 200 differ only on the claims where their answers differ, and the noise on the lead is about the square root of that count: if two prompts disagree on 9 claims, a lead of 3 is one standard error. Report differences in cases, treat four or fewer as no difference, and count the claims the two prompts disagree on before calling five a win.
Adjudication moves the ceiling
The 26 disagreements are adjudicated: both handlers and the champion, the Head of Claims Operations, go through them, she decides each one, and every decision that exposes an unwritten rule goes into the label instructions. Building the eval set works the meeting through. Afterwards every claim has one label reached on purpose, and the ceiling is no longer the raw 87%. It is the agreement a careful reviewer reaches with the adjudicated gold, working from the new instructions, and it is measured, not assumed.
Measure it with someone who was not in the room. A third senior handler, who did not label the 200 or sit in the adjudication, labels 100 of them blind from the revised instructions, and agrees with the adjudicated gold on 97: a ceiling of about 97%. The definition of done sets agreement at 95%, one noise band under it. The first run on the completed 200, at the week-4 gate, reads 185, 92.5%, with the high-value rule in code since week 3: one missed escalation and 14 false ones, so precision on escalate, as score the eval set defines it, is 39 of 53, 0.74, under the head of underwriting's 0.80 line as well. With the rule switched off the same run reads 182, 91%: 4 misses, precision 36 of 50, 0.72. At 185 the pilot is below the agreement gate, the precision line and the ceiling, so the five claims to the gate and the false escalations are real work, not noise.
The adjudicated ceiling works in both directions. A pilot reading 99% on adjudicated gold beats a careful person, and the usual cause is leakage: claims from the 200 used as worked examples, or read while tuning the prompt. The rule that prevents it: the 200 never enter the prompt, nobody writing the prompt reads them, and tuning uses a separate development set drawn the same way.
To the sponsor with the leaderboard, then: "That number is the model on a public task; ours is the pilot on your 200 claims, against the line you signed. Running the other model on them costs about 1r; I'll bring both numbers on Thursday." Say in advance what would change the decision: the other model wins if it agrees with gold on at least five more claims and misses no more escalations. The general mechanism is evals as the unit test of LLM work; this module applies it to one customer's reviewers and line.
In an interview
The deep-dive round asks "how do you know it works?" and the interviewer listens for four things: the customer's own cases, the customer's own labellers, a ceiling taken from how often those labellers agree, and differences stated in cases rather than points. In the role, the QA lead tests the same thing the first time you show her a number: she asks what it is a percentage of and who labelled it.
The mistake that loses points is reporting a score above the reviewers' own agreement as a success — "94%, better than your handlers agree with each other". It tells the interviewer the eval is measuring one labeller rather than the model, and that you did not notice.
Check yourself
1. The gold set is still handler A's labels; A and B agree on 87%. Prompt v4 reads 88% against A, prompt v5 reads 93%. Do you ship v5 into the shadow run?
Not on this evidence. 93% is above the 87% ceiling, so the gain may be fit to A; check first whether v5's new worked examples came from A's labelled cases. Adjudicate the 26 (90 minutes), re-run both prompts on the adjudicated gold, and ship v5 if it leads by more than four claims there.
2. The sponsor forwards a leaderboard on which another model is three points ahead of yours. Estimate what checking it costs, and say what result would make you switch.
200 claims × 0.005r ≈ 1r, plus an afternoon. Switch only if the other model agrees with gold on at least five more claims and misses no more escalations. Three points on a public task predicts nothing about the 200, and a switch resets the prompt work.
3. The third senior handler's blind re-label agrees with the adjudicated gold on 91 of 100, not 97. What happens to the 95% gate?
The ceiling is about 91%, so a 95% gate cannot be reached as written. Look at the nine: if they cluster on one rule the instructions do not cover, write the rule and re-measure this week. If they do not, the label is genuinely hard, and the gate moves to about 89%, with the sponsor told at the week-4 gate rather than finding out in week 8.