LLM-as-judge and its failure modes
Write a judge as binary criteria in a schema, correct for position, length and self-preference, and calibrate it on 50 human labels before trusting it.
On this page
A model judge is the grader for output no code can check: whether a reply answers the question, whether a summary is faithful to its source. It is a model call with a rubric, so it fails the way model calls fail, and it adds biases of its own that pull its verdicts in predictable directions. A judge's verdict is worth something once it has been measured against people, and not before. The judge has its own eval.
What a judge call looks like
The inputs are the rubric, the case input, the output under review and, where one exists, a reference answer. The output is a verdict in a schema, one field per criterion, each a reason and a pass or fail:
{
"answers_the_question": { "reason": "gives the refund window the customer asked for", "pass": true },
"policy_correct": { "reason": "states 60 days; the policy says 30", "pass": false },
"no_invented_facts": { "reason": "every figure appears in the ticket or the policy", "pass": true }
}
Binary criteria beat a 1-to-10 score. A score invites the judge to hover around the middle, a pass mark of 7 is a number nobody can defend, and an average hides which criterion failed. Three yes-or-no questions give a pass rate per criterion and a sentence to read when one fails. The reason comes before the verdict in the schema so the judge writes it first; a verdict written before its reasoning is a guess followed by a justification.
Four failure modes, after Zheng et al.
Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (opens in a new tab) (2023), measured how well model judges agree with people and named the biases that pull them away. Their strongest judge agreed with expert human preferences over 80% of the time, about as often as the experts agreed with each other. That is the encouraging result, and also the ceiling. The biases are why your judge, on your task, needs a number of its own.
Position bias. Shown two answers, a judge favours one slot — usually the first — whatever the content. Zheng et al. found that swapping the order changed the verdict on a substantial share of pairs for every judge they tested. The fix is mechanical: judge each pair in both orders, count a win only when both orders agree, and call the rest ties. That doubles the cost of pairwise judging.
Verbosity bias. A judge prefers the longer answer when the extra length adds nothing; Zheng et al. showed it by padding answers with restatements, and most of the judges they tested preferred the padded version. In your eval it looks like a pass rate that rises whenever a prompt change makes outputs longer. Test it directly — re-judge a sample trimmed to the old length with the content kept — and grade specific criteria ("states the refund window") rather than overall helpfulness, which length inflates most.
Self-preference. A judge may rate output from its own model family higher than a person would. Zheng et al. saw two of their three judges favour their own answers and said their data was too thin to settle it, which is a reason to measure it, not to assume it away. When the judge and the feature share a model, the eval is partly the model grading itself. Use a judge from a different family where you can; where you cannot, measure the gap on the calibration set below.
Drift after a model update. This one is not from the paper; it comes from running a judge in production. The judge is a model behind a name, and its provider updates it. Every judged pass rate then moves while nothing in the feature changed. Pin a dated judge version, record it in every result, and re-run the calibration whenever the judge's model or prompt changes. A judged pass rate that jumps with no change to the feature is a story about the judge until shown otherwise.
Pairwise versus absolute
An absolute judge grades one output against the rubric, pass or fail per criterion. A pairwise judge sees two outputs for the same input and says which is better. Pairwise is the easier question — people and judges are both more reliable comparing than scoring — and it is the right tool for choosing between two prompt candidates during development. But it produces no pass rate, it doubles with the order swap, and it says only which of two outputs is better, not whether either is good enough.
A regression gate needs a number that means the same thing next month, so the gate uses absolute pass-or-fail criteria. Pairwise belongs in the bake-off; absolute belongs in CI. The regression gates lesson is built on the absolute rate.
Calibrate on 50 human labels
Take 50 outputs the feature produced. Have the owner label each against the same rubric — 50 minutes at a minute each — and run the judge on the same 50. Report the two-by-two table, not one percentage.
The table above is a typical first result. The judge agrees with the owner on 42 of 50, 84%. It looks usable and is not: the owner failed 12 drafts and the judge caught 6. A judge that passes half of the human failures passes half of every regression it is meant to stop. Agreement hides this because passes dominate — a judge that said pass to everything would agree on 38 of 50, 76%.
So report two rates beside every judged pass rate: agreement, and the share of human-failed items the judge also failed. Then treat the judge prompt like any prompt, with the discipline of prompt design as interface design: read the six misses, tighten the criterion they share, re-run on the 50. Enrich the calibration set with failures — 20 to 25 of the 50 — because the catch rate is measured on the failures alone: on the 12 above, a 50% catch rate has a standard error of √(0.5 × 0.5 ÷ 12) ≈ 14 points.
Fifty labels is a rough instrument. At 84% agreement the standard error is √(0.84 × 0.16 ÷ 50) ≈ 5 points. That is enough to tell a usable judge from an unusable one and to notice when an update moved it, and not enough to choose between two judge prompts a point apart.
What the judge costs
A judge call reads the rubric, the input and the output — about 1.2k tokens for a support reply — and writes about 100 tokens of reasons and verdicts. A 200-case judged run is 240k × P_in plus 20k × P_out; pairwise with the swap doubles it. That is usually less than the feature run it grades. The real cost is the calibration labour, 50 minutes per judge change, and it is the best-spent hour in the eval.
A panel of three judges with a vote triples the run and catches disagreement between judges, not a blind spot they share; multi-agent patterns prices that trade, and the field version works the same choice for a customer deployment.
In an interview
"How would you evaluate a summariser?" is the usual prompt, and what is tested is whether a model judge arrives with a number attached. Say: binary criteria in a schema, reasons before verdicts; a judge from a different model family where possible; both orders for any pairwise comparison; and "calibrated on 50 owner labels enriched with failures, reporting agreement and the catch rate on human failures". Name the four biases and the fix for each in one breath.
The mistake that loses points is a judge with no calibration, or one whose agreement is quoted without the catch rate. Second: a 1-to-10 score averaged across cases, which nobody can turn into a gate.
Check yourself
1. A judge agrees with the owner on 46 of 50 items. The owner failed 5, and the judge failed 2 of those. Do you gate on it?
No. It catches 2 of 5 human failures, a 60% miss rate on exactly the items a gate exists to stop; agreement is high because 45 of 50 are passes. Rewrite the criterion the three misses share, enrich the calibration set to 20 failures, and re-measure. Gate on the judge once it catches at least 8 in 10.
2. Two prompts, judged pairwise in both orders on 100 inputs: A wins both orders on 40, B wins both on 25, and the other 35 flip with the order. Which prompt do you take?
Thirty-five verdicts out of 100 were decided by position, so they are ties. On the 65 decided pairs A leads 40 to 25, 62%, with a standard error near 6 points: about two standard errors, a lean towards A rather than proof. Take A if it costs no more; if A is the more expensive prompt, add 100 inputs before paying for it.
3. The judge's provider updates its model. The judged pass rate goes from 84% to 90% overnight, and the feature did not change. What do you do?
Treat the 6 points as the judge's until shown otherwise. Re-run the 50-label calibration on the new judge version; if agreement or the catch rate moved, re-judge the baseline with the new version before comparing anything. Then pin the judge version, so the next change happens on your schedule.