Exact, rubric and model-graded
Pick exact match, a human rubric or a calibrated model judge for each field of the output, and run the eval as a regression test on every prompt change.
On this page
Every output the pilot produces is graded one of three ways: compared exactly with a gold value, scored by a person against a rubric, or scored by a model applying that rubric. The three differ in cost by orders of magnitude and fail in different ways, and the choice is made per field of the output, not once per project. The triage record has fields of all three kinds.
The triage record, field by field
The pilot writes one record per claim, about 120 words, the one the prompt is a spec defines: the governing clauses, up to three clause ids from the retrieved set, each with a short verbatim quote; an exclusion flag; a route, fast_track, adjuster, fraud_review or needs_handler, with a decline reason when it is needs_handler; and a reason of at most 30 words, the rationale the handler reads in the claims form's 250-character note field. The escalation decision is derived from it: escalate when the route is fraud_review or needs_handler or the claim is over the high-value limit, otherwise approve. The decision is a label, graded by exact match against gold. The clauses are a set, graded by whether the clause the handler chose is among the three: the retrieval hit rate the kickoff wrote into the definition of done. Only the rationale is free text.
The common mistake is grading the whole record one way. Grade everything with a judge "because the output is text" and you pay a model to compare two strings, inheriting its biases on a field that never needed it. Grade everything by exact match and the rationales never match a reference paragraph, so nobody looks at them again.
Exact match: free, and only as good as its normaliser
Exact match costs nothing beyond generating the outputs, 200 × 0.005r ≈ 1r for a run of the pipeline. It runs in seconds and returns the same answer every time, which makes it usable on every change.
It goes wrong in two ways. The first is a comparison that is too strict: "4.2(b)", "4.2 (b)" and "s.4.2b" are one clause, and a grader that counts three reports a formatting difference as a retrieval failure. The fix belongs in the parser that reads the model's output, mapping every clause reference to the canonical id and refusing what it cannot map, as normalise the export does for dates.
The second is the wrong total. Accuracy over the decision hides the label that matters: with 160 gold approvals in the 200, a pilot that approves everything scores 80%. Report precision and recall on escalate and the count of missed escalations, as score the eval set implements. Four missed escalations is a number the head of underwriting can picture; an accuracy figure is not.
Rubric grading by people
The rationale has no gold string, so it is graded against a rubric of binary lines: names the governing clause; states the same decision as the record; cites no policy term absent from the retrieved section; contains no personal data beyond the claim number. Binary lines beat a 1-to-5 score: two reviewers agree more often on "does it name the clause" than on "rate the clarity", and a failed line points at a fix, where 3 out of 5 points at nothing.
The rubric has its own ceiling. When the two senior handlers graded the same 100 rationales, they agreed on 92% of lines, so no grader of this rubric, person or model, can be held to more than about 92% agreement with a human grade.
People are the right grader at the gates and the wrong one in the loop. A rationale takes about 2 minutes, so 200 take about 6.7 reviewer-hours: affordable at the gates in weeks 4, 6 and 8, but at three prompt changes a day it would be 20 reviewer-hours a day.
Before handing a line to any grader, ask whether code can check it. "States the same decision as the record" is exact match between two fields of one output. "No personal data" is a string check of the rationale against the claim's own personal fields. Two lines go to code, and the judge keeps the two that need reading: whether the named clause governs, and whether any cited term is invented.
Model-graded: the judge and its biases
A judge model reads the claim, the retrieved section, the whole record with its quotes and reason, and the rubric, and writes a verdict per line. Its cost per rationale, at the course's 0.75 words per token: input of 1,600 words of claim and note, 900 of policy section, 120 of record and 400 of rubric and instructions, 3,020 words ≈ 4,000 tokens, 0.0040r; output of about 75 words ≈ 100 tokens at 4r, 0.0004r. About 0.0044r a rationale, 0.9r for the 200: less than one run of the pipeline, against 6.7 reviewer-hours.
Judging LLM-as-a-Judge (Zheng et al.) examined three biases in model judges; it measured the first two and could not establish the third on its data. Each has a standard countermeasure:
- Position. Asked which of two answers is better, a judge often favours one position: the strongest judge in the paper gave the same verdict after a swap only 65% of the time. Grade each pair in both orders and count only preferences that survive the swap, or grade each rationale against the rubric alone, as the triage eval does.
- Length. Longer answers tend to score higher. Use binary lines that length cannot satisfy, and check the reason's 30-word, 250-character limit in code.
- Self-preference. A judge may rate text from its own model family higher; the paper saw signs of it but could not confirm it, so treat it as a risk. Use a judge from a different family from the pilot's, and let calibration catch what remains.
Calibration comes before trust. Run the judge on the 100 rationales the handlers graded and compare line by line with the adjudicated human grade. A judge within two points of the 92% human ceiling can grade in the loop. A judge at 82% only flags rationales for a person, and the line it fails on is the first candidate for code. Re-calibrate whenever the judge's prompt or model changes. The report all this prevents is "the judge gave v6 4.6 out of 5": a scale no two people agree on, a same-family judge, no calibration number. The general treatment is LLM-as-judge and its failure modes.
The eval as a regression test
The 200 cases, the gold labels, the label instructions, the grader code and the judge prompt are versioned as one thing; a change to any of them is a new baseline. Every prompt change runs the eval before it reaches the shadow or assisted queue:
- The CI runner sends the 200 claims through the candidate prompt: about 1r.
- Exact match grades decision and clause: agreement, precision and recall on
escalate, missed escalations, all per segment. - Only if those are inside the band does the judge grade the two rationale lines: about 0.9r.
- The runner compares with the baseline, and blocks the change if it adds even one missed escalation, or drops agreement by more than four claims overall or two on commercial's 40.
- People grade the full rubric at the gates, not in the loop.
The block rule is asymmetric on purpose: a four-claim drop in agreement is inside the noise band, but one added missed escalation moves the missed rate by 2.5 per 100 on 40 gold escalations. The cost is about 2r a change. Ten changes a week is about 20r, more than the shadow run's 12.5r a week at 500 claims a day and two-thirds of the 5 × 6r = 30r a week the full floor will cost; the alternative is a week-6 gate failed by a week-5 change nobody ran the 200 on. Regression gates covers the general form; here the gold, the rules and the gate all came from the customer's reviewers, so a regression is a regression in their terms.
In an interview
The deep-dive round asks "how would you grade this?" about an output with mixed fields. The interviewer listens for a choice per field, a line moved from judge to code, the judge's calibration number, a named bias with its countermeasure, and a rule for what blocks a change. In the role, the QA lead notices whether a prompt change can reach her handlers without passing through the 200.
The mistake that loses points is a judge nobody checked against people: "we use model-graded evals", with no agreement figure against a human grade and a judge from the same family as the model being graded.
Check yourself
1. A prompt change lifts agreement from 185 to 188 of 200 and adds one missed escalation. Merge it?
No. Three claims is inside the four-claim noise band, so the gain may not exist, and the added missed escalation is a block by rule: 2.5 per 100 on the line the head of underwriting signed. Find the newly missed claim and why before trying again.
2. On the 100 human-graded rationales, the judge agrees with the handlers on 84% of lines; the handlers agree with each other on 92%. Do you put it in CI?
Not as a grader. Find the line it disagrees on. If it is "cites no invented policy term", add a code pre-check that flags any cited term not found verbatim in the retrieved section, leave the judge only the paraphrased citations, and re-calibrate. Until it reaches about 90%, the judge only flags rationales for a person.
3. For the two weeks before the week-6 gate the team expects three prompt changes a day. Estimate the eval's cost with the full loop on every change, and with the judge only on the one merge candidate a day.
Full loop: 3 × 2r × 10 working days = 60r. Exact match on every change and the judge once a day: 3 × 1r × 10 + 0.9r × 10 ≈ 39r. The 21r saved is three and a half days of full-floor traffic, about eight of the shadow run's, and little is lost: routing tweaks rarely move the rationale lines, and the merge candidate is still judged. Take the cheaper loop.