Building the eval set
Draw 200 claims by segment, write the label rules, run two blind labellers, and turn the 26 disagreements into rules the gold set and the prompt share.
On this page
The triage pilot's eval set is 200 claims drawn from history, labelled twice by the customer's senior handlers, with every disagreement settled into a written rule. Each step has a cheap version that leaves the 95% gate measuring nothing: a convenient sample, labelling without instructions, and disagreements quietly dropped. Labelling the set below started in week 1, while the prototype ran on the one-off export, and the set was signed at the week-4 gate.
Where the 200 come from
The convenient sample is the one already to hand: the champion's examples, or last week's claims. The demo that ran on the sample data shows what the first costs: 29 of 30 on chosen cases, 71% on the next 1,100 live ones, and a sponsor who had already told the board 97%. Last week's claims fail more quietly: one week is one weather pattern, and a storm week can be 40% property water damage.
Draw from the backfill instead. Getting the data out ran the extractor once over 65 working days, 78,000 rows, for this purpose. Three filters come first: only claims whose triage is complete, since an untriaged claim cannot be labelled against the file the handler saw; none of the claims used to build the prototype or to choose the prompt's worked examples; and none that either labeller triaged, so nobody labels their own claim from memory.
Stratify by segment
A simple random draw of 200 gives each segment its share on average, and not reliably. Commercial is 20% of claims, so it averages 40 of 200, but the count varies by √(200 × 0.2 × 0.8) ≈ 5.7 from draw to draw: anywhere from 29 to 51 at two standard deviations. A draw with 29 commercial claims under-weights the hardest segment, and the blended agreement figure looks better than the floor will find it.
Stratification removes the chance. Split the backfill by segment, then draw at random within each: 110 motor, 50 property and 40 commercial, the live proportions of 55%, 25% and 20%. Each count is fixed before anyone labels, so each segment's figure has a known resolution: on 40 commercial claims one case is 2.5 points, and a reading near 95% carries about ±7. That is enough to see a prompt change cost commercial cases, and too little to gate the segment, which is why crawl, walk, run reads the per-segment gates on the shadow run's 1,000 commercial claims.
When a small segment has its own gate, give it a floor rather than its share, say 60 commercial instead of 40, and reweight the overall figure to the live mix: overall = 0.55 × motor + 0.25 × property + 0.20 × commercial. An unweighted figure on an oversampled set over-counts the oversampled segment, and the report says which of the two it shows.
One thing cannot be stratified: the gold label, since nobody knows it before labelling. Escalations are about 20% of claims in the historical escalation field, so the 200 hold roughly 40 gold escalations, and on 40 one miss is 2.5 per 100. The head of underwriting's line, at most 2 missed escalations per 100, therefore means zero misses on this set, and zero misses in 40 is still consistent with a true rate as high as 3 ÷ 40 ≈ 7.5 per 100, by the rule of three. Certifying that line needs about 150 gold escalations (3 ÷ 150 = 2 per 100), 110 more than the set holds: a supplement of about 120 claims the historical field marks as escalated, assuming about 9 in 10 of them survive labelling, labelled the same way and scored for recall only. The historical field may choose which claims to label; it is never the label. At 2 × 11 minutes a claim that is about 44 reviewer-hours, the sponsor's decision, put to him in those hours.
The label instructions
One page, written before anyone labels, versioned. It states the question being labelled ("would a senior handler need to see this claim before anyone acts on it?"), the two allowed answers, every rule already known, and what to do when unsure: choose, and mark the case unsure so it is adjudicated first. It also states what the labeller must not see. The historical escalation field is hidden, because a labeller who can see it copies it; the other labeller's answer is hidden; and nobody labels with the pilot's suggestion on screen.
Labelling is the expensive step. A label is a full triage, about 11 minutes, so 200 claims × 2 handlers × 11 minutes ≈ 4,400 minutes, 73 reviewer-hours; at two and a half hours a day each, 15 working days. That is why labelling starts in week 1, and why the scoping doc names the QA lead as owner of the set.
174 agreed, 26 did not
The handlers agreed on 174 of 200, 87%. Three quick ways of handling the 26 each damage the set.
Dropping them leaves the 174 claims on which two people found the answer obvious. Agreement on that set flatters the pilot by exactly the cases where judgement is tested, and the gate is read on a set easier than the floor.
Breaking ties by seniority, or by whichever handler the champion trusts more, turns the gold set into one person's labels on the hard cases: the noise an eval is not a benchmark warns about, concentrated where it matters most.
Deciding them yourself puts your reading of their policy into their gold set.
Adjudicate instead: ninety minutes, run by the claims QA lead who owns the set, with both handlers and the champion, the Head of Claims Operations, the 26 files on screen, the champion deciding each one and you writing down why. For the triage pilot the 26 fell into three groups:
- 9 slips. One handler misread an amount or missed an attachment. Corrected; no rule.
- 13 instruction gaps, which collapsed into three rules: a claim above the high-value band escalates whatever else is true (5 cases); a claim whose governing clause sits in an endorsement escalates (5); a second claim on the same policy within 12 months escalates (3).
- 4 genuinely ambiguous files. The champion ruled that when the file does not settle it, the claim escalates, and that became the fourth rule.
Then the step most teams skip: apply the four rules to the 174 the handlers agreed on. Two were claims above the high-value band that both handlers had approved. Agreement is not correctness, and both labels flipped. The final set has 200 labels, 40 of them escalate.
The rules go to three places. Into label instructions v2, so the next labeller applies them. Into the pilot: the high-value rule as a line of code before the model is called, the others into the prompt written as a spec, so the claims they cover come out routed fraud_review or needs_handler, the two routes that escalate. And into the field note, because a rule nobody had written down is a property of the customer's process that the next deployment will meet too.
Measure the new ceiling
A third senior handler, who neither labelled the 200 nor sat in the adjudication, labels 100 of them blind from the v2 instructions, and agrees with the adjudicated gold on 97. That 97% is the ceiling the 95% gate is read against. Had it come back at 91, the instructions would still have a gap, and the week-4 gate, before the shadow run goes to 500 a day, is the time to find it. The general mechanism, a golden set and its upkeep, is in building the golden set; what this lesson adds is that the labels, the rules and the ceiling all belong to the customer.
In an interview
"Walk me through building the eval" is a deep-dive staple. The interviewer listens for where the cases come from (history, not the champion's folder), how they are split (by segment, with counts), who labels and how (two customer reviewers, blind, from written instructions), and what happens to the disagreements. In the role, the QA lead who owns the set notices one thing above the rest: whether the 26 were settled by the customer's own people, or by you.
The mistake that loses points is dropping the disagreements and reporting agreement on the rest. It removes the 13% of claims where judgement is tested and reports the pilot's score on the easy 87%.
Check yourself
1. The draw is done and holds 31 commercial claims, not 40. Re-draw, top up, or keep it?
Top up: draw 9 more commercial claims at random from the commercial stratum, 9 × 22 minutes ≈ 3.3 reviewer-hours. Re-drawing throws away labelling already done; keeping 31 leaves one commercial case at 3.2 points and under-weights the hardest segment, which flatters the blended figure.
2. The sponsor can give 40 reviewer-hours, not 73. How many claims do you label, and how?
Not 109 claims double-labelled. Double-label 50 first (50 × 22 minutes ≈ 18 hours), adjudicate them and write v2 of the instructions; then single-label 120 more from v2 (120 × 11 minutes = 22 hours). That is 170 claims in 40 hours, with agreement measured and the rules written. Tell the sponsor commercial will hold about 34 claims, so its figure moves about 3 points a case.
3. Twenty-one of the 26 disagreements are handler B approving where handler A escalated. What do you do differently before the adjudication meeting?
Treat it as systematic, not noise: B is applying a narrower reading of when to escalate, and one or two rules probably cover most of the 21. Put those cases first in the meeting and ask the champion which reading is the policy; then apply the resulting rule to the 174 agreed claims, where A may have followed B's reading on some.