Evals with the customer's definition of good · Grading
Score the eval set
Write the scorer that returns precision, recall and missed escalations, then decide whether a run that passes precision and fails recall passes the pilot.
codeeasyExercise · 3 min · 43 of 64
On this page
The eval harness needs one function, run on every prompt change, that turns 200 gold labels and 200 predictions into the numbers the definition of done is written in. It is fifteen lines; the decision it feeds is the hard part.
The claims-triage pilot decides, for each claim, escalate or approve. An escalated claim goes to a senior handler, at about 4 minutes of senior time; an approved one goes straight to the desk, so a missed escalation is a high-value or disputed claim decided without a senior seeing it. Two senior handlers labelled 200 claims escalate or approve, and agreed on 174. The 26 disagreements were adjudicated, so the gold set has one label per claim, 40 of them escalate. A candidate prompt labelled the same 200. The head of underwriting had two lines added to the definition of done for this decision, by an amendment the scoping doc's signatories countersigned: at most 2 missed escalations per 100 gold escalations, and precision on escalate of at least 80%. The harness that runs the 200 on every prompt change, described in exact, rubric and model-graded, needs its scorer.
Compute precision, recall and the missed-escalation count for the pilot's output; decide whether it passes; and say what you tell the customer if it passes precision and fails recall. The scorer's contract:
def solve(gold, pred, positive="escalate"):
"""(precision, recall, missed) for the positive label, rounded to 3 dp."""
Fix the positive label before counting: escalate, because the costly error is a claim approved that a senior should have seen. One pass over the paired labels counts true escalations, false escalations (escalated, gold approve) and misses (approved, gold escalate). Precision is true over predicted escalations, recall is true over gold escalations, and missed is the miss count, an integer the customer can picture. The scorer never computes accuracy, and it refuses lists of different lengths rather than zipping them short and silently dropping claims. An empty denominator scores 0.0, said in the code: a pilot that escalates nothing must not pass a precision gate on no evidence.
On the candidate's run, 40 gold escalations, 36 caught, 4 missed, 42 escalated in all: precision 36 ÷ 42 = 0.857, recall 0.90. The head of underwriting's two lines are independent gates, and independent of the agreement line. Precision passes with six points to spare. Recall is 10 missed per 100 against a limit of 2, so the pilot fails, whatever its precision. On this set the gate is stricter than it reads: one miss in 40 is 2.5 per 100, so passing needs zero.
To the customer: "Four claims a senior should have seen were approved, so the run fails the line you signed. The fix is the threshold and a rule, not the labels: anything above the high-value band escalates regardless of the model, the escalation threshold comes down to spend some of the precision headroom, and we re-run the 200 on Thursday." Re-labelling the four until they pass would change the test, not the pilot.
"Report accuracy, correct over 200." Loses because with 160 gold approvals, approving everything already scores 80%, and this candidate reaches 190 of 200, 95%, exactly the agreement line, while missing 4 of 40 escalations: accuracy is one line of the definition of done, and alone it hides the line the head of underwriting signed.
"Return 1.0 for precision when nothing is escalated, since no escalation was wrong." Loses because a prompt change that breaks the escalate branch entirely then reads as perfect precision; the gate passes on no evidence, and only recall catches it, if anyone is reading recall.
"Ask the handlers to look at the four misses again; the gold might be wrong." Loses because the gold was adjudicated before the run; re-opening labels only after the pilot fails them grades the set to the answer, and the head of underwriting will ask why only those four were re-opened.
Python 3 · standard library30 lines · 8 of 8 tests passing
"""Score the eval set — precision, recall and missed escalations for one positive label."""
def confusion(gold, pred, positive):
"""(true positives, false positives, misses) over two aligned label lists."""
tp = fp = missed = 0
for g, p in zip(gold, pred):
if p == positive and g == positive:
tp += 1 # escalated, and gold says escalate
elif p == positive:
fp += 1 # escalated a claim gold approved: a senior's few minutes
elif g == positive:
missed += 1 # approved a claim gold escalated: the costly error
return tp, fp, missed
def rate(num, den):
"""num / den to 3 dp. An empty denominator scores 0.0, so no gate ever passes on no evidence."""
return round(num / den, 3) if den else 0.0
def solve(gold, pred, positive="escalate"):
"""(precision, recall, missed) for the positive label, rounded to 3 dp."""
# zip() would silently truncate to the shorter list and drop claims from the score.
if len(gold) != len(pred):
raise ValueError(f"{len(gold)} gold labels against {len(pred)} predictions")
tp, fp, missed = confusion(gold, pred, positive)
# Precision: right escalations among predicted ones. Recall: right ones among gold ones.
# Accuracy is never computed: with 160 gold approvals, approving everything scores 80%.
return rate(tp, tp + fp), rate(tp, tp + missed), missed
The cases that ran
TESTS = [
((["escalate", "approve", "escalate", "approve"],
["escalate", "escalate", "approve", "approve"]), (0.5, 0.5, 1)),
((["escalate", "escalate", "escalate", "approve", "approve"],
["escalate", "escalate", "escalate", "approve", "approve"]), (1.0, 1.0, 0)),
((["escalate", "approve"], ["approve", "approve"]), (0.0, 0.0, 1)),
# A candidate prompt's run on the 200: 40 gold escalations, 36 caught, 4 missed, 6 false escalations.
# Precision 36/42 passes the 0.80 line; recall 0.90 is 10 missed per 100 and fails.
((["escalate"] * 40 + ["approve"] * 160,
["escalate"] * 36 + ["approve"] * 4 + ["escalate"] * 6 + ["approve"] * 154), (0.857, 0.9, 4)),
# Escalate everything: recall is perfect, and precision falls to the share of gold escalations.
((["escalate", "approve", "approve", "approve", "approve"], ["escalate"] * 5), (0.2, 1.0, 0)),
# No gold escalations: recall has no denominator and scores 0.0; nothing can be missed.
((["approve", "approve"], ["escalate", "approve"]), (0.0, 0.0, 0)),
# Another positive label, passed explicitly: the phase-2 fraud flag.
((["fraud", "clear", "fraud", "clear"], ["fraud", "fraud", "fraud", "clear"], "fraud"), (0.667, 1.0, 0)),
# An empty set scores nothing and raises nothing.
(([], []), (0.0, 0.0, 0)),
]
