The role, its neighbours and the interview · The interview

The deep-dive — defend the eval

Design the eval you would defend when a senior engineer challenges your 96%: what the gold set is, what the number is measured against, and what you add.

designmediumExercise · 4 min · 62 of 64

On this page

The deep-dive interviewer has found the three soft spots in your best number. The round is scored on whether you can say which challenges are real and design the fix in reviewer-hours, which is what the typical loop means by depth on one component.

The situation

In the deep-dive you chose the claims-triage pilot: 1,200 claims a working day, a definition of done of 95% agreement with the adjudicated label on a 200-case gold set. You say the pilot reached 96%. The interviewer, a senior engineer, pushes back: "Two handlers labelled those 200 and agreed on 174. You tuned the prompt against those 200 for three weeks while they were being labelled. Commercial is 20% of volume and you never mentioned it. Why should I believe 96%?" Everything she says is true. In this telling of the pilot, the 200 were drawn uniformly at random from one week of claims, not stratified as in building the eval set. The 26 disagreements were adjudicated by the champion, the Head of Claims Operations, and the gold label is the adjudicated one; a blind re-label of 100 agreed with it on 97. Labelling a claim takes a reviewer about 11 minutes, and the two senior handlers can each give two and a half hours a day, 25 hours a week between them. The shadow gate is two weeks away.

The question

Design the eval you would defend in the next ten minutes: what the 200 cases are, what 96% is measured against, and what you add so that a prompt tuned on the set can still be trusted at the gate.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Take the three challenges in order of the damage they do.

The 87% is not the ceiling. The 26 were adjudicated, so the ceiling is a careful reviewer's agreement with the adjudicated gold, which the blind re-label put at 97 of 100; 96 sits under it. Raw inter-rater agreement caps a score only when the gold was never adjudicated. Say that first: it has a clean answer.

The tuning is the real problem. Three weeks of prompt changes scored on the same cases make them a development set, and 96% on a development set is an upper bound. Add a held-out set the prompt author never sees: 100 claims from the latest week, double-labelled and adjudicated as before. That is 100 × 11 minutes × 2 ≈ 37 reviewer-hours, inside the 50 the handlers can give in two weeks. On 100 cases the standard error near 95% is about 2 points, so a holdout at 92 or above is consistent with the 96; below 92, the 96 was the development set talking.

Commercial is a sampling problem: uniform sampling gives it about 20 of 100, where one case moves it 5 points. Stratify the holdout to 30 commercial cases and weight the overall figure back to volume. Leave the segment's gate to the shadow run, 95% on its 1,000 commercial claims: 30 cases carry about ±8 points.

Grading stays exact match: the label is categorical, and a model-graded judge would add a second source of error. The 200 run on every prompt change as a regression test; the holdout runs once, at the gate, so it stays unseen.

What would reverse the defence: the holdout below 92, or commercial below 95% in the shadow run.

Rubric

separates raw inter-rater agreement from agreement with the adjudicated gold, and says which ceiling applies
names the tuned set as a development set and adds a held-out set the prompt author never sees
sizes the holdout in reviewer-hours against the hours available before the gate
puts a noise band on the holdout and states the reading that would sink the 96
stratifies commercial in the holdout, weights the overall figure back to volume, and gates the segment on a sample large enough to read
keeps exact-match grading and separates the per-change regression run from the one-time holdout

Where answers go wrong

"96% clears the 95% line, so it passes." Loses because it answers none of the three challenges; the interviewer asked why to believe the number, not whether it is above the line.

"Have a third reviewer re-label all 200 to raise agreement." Loses because adjudication already settled the gold, and 200 claims at 11 minutes is 37 reviewer-hours, leaving 13 of the 50 and no room for the holdout, spent on the one challenge that was already answered while the tuning problem stays open.

"Grade 2,000 claims with a model-graded judge for a bigger sample." Loses because the label is categorical and exact match costs nothing; the judge's errors stack on the prompt's, and the 2,000 still have no gold labels for the judge to be checked against.