Evals with the customer's definition of good · Grading

Judge or human for 5,000 cases

Estimate the reviewer-hours and the judge's cost, as a multiple of r, to grade 5,000 rationales before sign-off, and decide who grades which line.

estimatemediumExercise · 3 min · 45 of 64

On this page

Before handlers start reading the pilot's rationales, the QA lead wants every one of them graded. The estimate decides whether that is possible by the sign-off date, and who does which part.

The situation

End of the triage pilot's shadow run: weeks 5 and 6 at 500 claims a day produced 5,000 triage records, each with a rationale of at most 30 words for the claims form's 250-character note field, unread by any handler. Walk, from week 7, keeps the note blank until the QA lead has all 5,000 graded on the four-line rubric: names the governing clause; states the same decision as the record; cites no policy term absent from the retrieved section; contains no personal data beyond the claim number. A senior handler grades one in about 2 minutes. Two handlers can give 2.5 hours a day each, and sign-off is due in 10 working days. Grading the same 100 rationales, they agreed on 92% of lines, and about 8% of rationales failed some line. A judge model would read the claim and adjuster note (1,600 words), the retrieved section (900), the whole record (120) and 400 words of rubric and instructions, and write about 75 words. Assume 0.75 words per token, input at r per million tokens and output at 4r; a claim costs about 0.005r through the pipeline.

The question

Estimate the reviewer-hours and the judge's cost to grade all 5,000 by sign-off, and decide who grades which line.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

People alone cannot finish. 5,000 × 2 minutes ≈ 167 reviewer-hours; at 2 × 2.5 = 5 hours a day, 33 working days against 10. The price of an hour never enters: the calendar decides.

Two lines need nobody. "Same decision as the record" is exact match between two fields; "no personal data" is a string check of the rationale against the claim's own personal fields. Both run on all 5,000 in seconds.

The judge takes the other two. Input: 1,600 + 900 + 120 + 400 = 3,020 words ≈ 4,000 tokens, 0.0040r. Output: 75 words ≈ 100 tokens at 4r, 0.0004r. About 0.0044r a rationale, so 5,000 × 0.0044r ≈ 22r, about nine days of the pilot's 500 × 0.005r = 2.5r a day.

Only after calibration, though: on the 100 rationales the handlers already graded it must agree with them on at least 90% of lines, two points under their own 92%, and it must come from a different model family from the pilot's.

People go where judgement is. About 8% failing is 400 rationales to confirm, 400 × 2 minutes ≈ 13.3 hours, plus a random 100 of the judge's passes to measure what it misses, 3.3 hours: about 17 reviewer-hours, under 4 working days.

So: code for two lines, a calibrated judge for two, people on the fails and a sample of passes, about 22r and 17 hours. If the judge calibrates below 90% on the clause line, that line goes to people on a 500-rationale sample, another 17 hours, and the QA lead is told now that she signs off on a sample.

Rubric

converts 5,000 rationales to reviewer-hours and working days, and compares them with the 10 days
moves the lines code can check out of both the judge's and the people's work
prices the judge per rationale from word counts, input and output separated, ending in a multiple of r
makes calibration against the handlers' grades a precondition, with a threshold tied to their 92%
sends people to the judge's fails and a sample of its passes, with the hours
says what happens if the judge does not calibrate

Where answers go wrong

"People; the QA lead won't trust a model." Loses because 167 hours at 5 a day is 33 working days against 10; sign-off slips by about five weeks to honour a doubt that the calibration exists to answer.

"The judge for all 5,000; 22r and done." Loses because an uncalibrated judge's pass is unverified and nobody reads its fails, so the sign-off rests on a grader no one has checked against a person.

"Convert both to currency and compare." Loses twice: the rate belongs to the contract, so a currency figure is a guess or a disclosure, and price was never the question, since the people-only option misses the date at any price.