Evals, observability and guardrails · Evals

The judge that agreed with itself

Take apart a model judge validated only against itself, tie each bias to a fact in the data, and say what has to happen before its 94% is reported.

critiqueeasyExercise · 3 min · 47 of 77

On this page

A judge that gives the same verdict twice has shown that it is consistent. Whether it is right is a different measurement, and this team has not made it.

The situation

A team grades the drafts of its customer-support reply drafter with a model judge. The judge comes from the same model family as the drafter, scores each draft from 1 to 10 for helpfulness, and passes a draft at 7 or above. To validate it, the team ran the judge twice over the same 200 drafts and got the same pass or fail on 97% of them; the design doc calls the judge "validated". No person has labelled a draft.

Last month a prompt change made the drafts longer, from about 120 words to about 220, and the judged pass rate rose from 88% to 94%. The team also compares candidate prompts pairwise, always showing the current prompt's draft first. The quarterly review is next week, and its slide reads "94% of drafts judge-validated". The lesson this practises is LLM-as-judge and its failure modes.

The question

Find the defects in how this judge was validated and used, and say what the team must do before the 94% goes on a slide.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Self-agreement measures repeatability, not correctness. A judge that is consistently wrong agrees with itself; 97% says the verdicts are stable and nothing about whether they are right. The judge has no eval.

Three biases from Zheng et al. are live, each with a confound in this data. Verbosity: the pass rate rose 6 points in the month the drafts got 80% longer, so nobody can tell better drafts from longer ones. Self-preference: the judge shares the drafter's family, so the eval is partly the model grading itself. Position: the incumbent's draft is always shown first, so every pairwise comparison leans towards keeping it.

The score is a fourth defect. A 1-to-10 helpfulness score with a pass mark of 7 cannot say what failed, and helpfulness is the criterion length inflates most.

Before the slide, replace the score with three binary criteria — answers the question, states the policy correctly, invents nothing — with reasons written before verdicts. Have the support lead label 50 drafts against them, about 50 minutes, enriched to 20 failures, and report agreement and the share of the lead's failures the judge also failed. Test length directly by re-judging 30 drafts trimmed to 120 words with the content kept. Run pairwise in both orders and count only consistent wins. Use a judge from another model family, or measure the self-preference gap on the 50.

The 94% can stand if the judge catches at least 8 in 10 of the lead's failures and the trimmed drafts score as the long ones did. If the trimmed drafts drop, the 6 points were length.

Rubric

distinguishes self-agreement from agreement with people, and says the judge has no eval
ties verbosity, self-preference and position bias each to a fact in the situation
rejects the 1-to-10 pass mark in favour of binary criteria, with the reason
specifies a human calibration set with its size, its enrichment and the two rates to report
proposes a direct length test and a both-orders pairwise run
names the evidence that would let the 94% stand

Where answers go wrong

"97% agreement is high, so the judge is fine." It confuses consistency with accuracy. The reviewer is checking whether the candidate knows a judge needs human labels before its number means anything.

"Use a stronger model as the judge." A stronger judge still has position and length bias and still has no calibration number. The defect is the missing eval, not the model.

"Average the 1-to-10 scores instead of using a pass mark." It moves the problem without fixing it: an average of a length-biased score is still length-biased, and no one can gate a release on 7.4.