Harnesses and multi-agent workflows · Multi-agent

A three-judge panel for a two-cent task

Find what a three-judge panel actually catches on a two-cent generation task, cost it per day, and replace it with a cheaper gate that catches the real error.

critiquemediumExercise · 3 min · 65 of 77

On this page

A panel of three judges sounds like rigour. Priced against the task it guards and measured against human labels, this one costs twice the generation and misses the error it was built to catch.

The situation

An online retailer generates product descriptions from catalogue data, 20,000 a day. Each generation reads about 4,000 tokens and writes about 500, roughly two cents. To gate quality the team added a panel: three calls to the same judge prompt at temperature 0.7, each reading the description, the product data and a 1,500-token rubric, about 3,800 tokens in and 150 out. A majority vote decides pass or fail, and a failed description is regenerated.

On 100 human-labelled descriptions, a single judge agrees with the humans 88 times and the panel 90 times. Of the panel's 10 disagreements, 8 are long descriptions it passed although a required spec field, dimensions or material, was missing. Rerunning one judge five times on the same item flips its verdict on 6% of items.

Course constants: P_in = $3 and P_out = $15 per million tokens.

The question

Find the defect in the panel, with its daily cost against the generation's, and say what you would replace it with.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Generation: 4,000 × $3/M + 500 × $15/M ≈ $0.012 + $0.0075 ≈ $0.02. One judge: 3,800 × $3/M + 150 × $15/M ≈ $0.014. The panel is $0.041 per description, twice the task it guards: about $820 a day against $390 for generation.

It buys 2 points of agreement, and the data says why so few. A 6% flip rate is the inconsistency a vote can fix, and a flipping item is right on most runs anyway, so the vote recovers only part of the 6: here, 2. The 8 remaining misses are systematic: three samples of one model with one prompt share its preference for long descriptions, so they agree, wrongly. That is the shared blind spot in multi-agent patterns.

Those 8 are not a matter of judgment either. "A dimension present in the data must appear in the description" is a check code can run: read the product data, look for each required field's value in the text, fail on a miss. No tokens, every time.

Replacement: the code check for required fields first, then one judge at temperature 0 with a checklist of the subjective criteria (tone, no claim beyond the data, length), each answered with a quote. About $0.015 per description with the extra evidence tokens, roughly $300 a day, saving about $520. Recalibrate on the same 100 labels: the 8 field misses should now fail in code, and agreement should match or beat the panel's 90.

What would reverse it: a flip rate near 20% with disagreements scattered rather than clustered. Then inconsistency dominates, and temperature 0, then a panel, is the fix.

Rubric

prices one generation, one judge and the panel per item and per day, with input and output separate
reads the 88-to-90 gain as the random part of the error and the 8 clustered misses as systematic
uses the rerun flip rate to separate inconsistency from bias
moves the required-field rule into a code check
proposes one checklist judge at temperature 0, costs it, and says how it will be calibrated
states the evidence that would make a panel the right fix

Where answers go wrong

"Add two more judges for a panel of five." Five samples of the same judge agree more confidently on the same 8 misses, at five times the cost of one judge, more than three times the generation itself.

"Use a stronger model as the judge." It may shift the bias, but the main miss is a rule code can check, and the change raises the cost of the most expensive part of the pipeline. The critique has to find the error class, not buy a bigger judge.

"Drop the panel; 88% is fine." It saves the money and keeps shipping the descriptions with missing dimensions. A critique that only removes cost has fixed half the defect.