Running it: evals, cost and security · Evals and monitoring

An eval set for bookkeeping prep

Design the eval set that decides whether bookkeeping prep may move to a new model: cases, labels, grader, and an acceptance line a cautious model cannot game.

designhardExercise · 4 min · 59 of 75

On this page

The model under the agency's bookkeeping prep is being retired, and its replacement has to be chosen before the accountant finds the difference in a filed return. The set built now gates this change and every one after it, as what working means sets out.

The situation

Bookkeeping prep categorises 600 bank transactions a month, about 140 a week, into a chart of 40 accounts, 12 of which carry 90% of the volume. Each transaction arrives from the bank feed as date, amount, counterparty and memo. The automation writes a draft account and a confidence and never posts: only the external accountant's signature, by the fifth working day, makes a month final. Anything under its threshold goes to a review queue the operations manager works. The accountant has signed the last three months: 1,800 transactions, each with its final account.

The operations manager, who keeps the books, categorises a transaction by hand in about a minute and reviews a queued one in about 2. A mis-post, a wrong draft the accountant signs, costs about $20 to find and correct later, and more if it reaches a filed tax return. Transfers between the company's own accounts (about 15 a month), tax payments (3) and payroll runs (2) must always reach the queue, never be drafted to an expense account. The model version in use is retired in six weeks. Assume staff time at $30 an hour and the founder's at $100.

The question

Design the eval set that gates the replacement model and every change after it: how many cases, drawn how, labelled by whom, graded how, and the acceptance line.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

150 cases. 120 stratified from the 1,800 signed transactions: 90 across the 12 large accounts in proportion to volume, 30 spread over the long tail. Then 30 targeted: 10 own-account transfers, 6 tax payments, 4 payroll runs — the 20-case must-pass slice — and 10 transactions the accountant corrected last quarter, the hard slice.

Labels. The signed ledger is the owner's label, already paid for. The founder checks only the 30 targeted cases against the accountant's notes: about an hour, $100.

Grader. Exact match on the account code: every case has one right answer, so no model judge. A case sent to review counts as queued, neither right nor wrong.

Acceptance line, on the same 150, compared case by case with the current model:

at least 97% of the cases it drafts are correct
at most 15% queued, so review stays near 600 × 15% × 2 minutes = 3 hours a month
zero drafts in the must-pass slice: all 20 reach the queue

Precision and queue share are read together, because a model that queues everything scores 100% on what it drafts. At 97% on about 128 drafted cases the standard error is √(0.97 × 0.03 ÷ 128) ≈ 1.5 points, so a 3-point gap between models is two standard errors; a close result gets a repeat run, not a guess.

Upkeep. Re-run on every prompt or model change and whenever a new bank feed is connected, since its memo format is new input. Add the accountant's corrections monthly. If more than one mis-post a quarter reaches a filing, widen the must-pass slice to its pattern.

Rubric

draws from the signed ledger, stratified, and fills the rare costly transactions to a floor instead of leaving them to a random draw
uses the accountant's signed accounts as labels and costs the labelling that remains
grades by exact match on the account code, with no model judge
sets an acceptance line with draft precision, a queue-share cap and a zero-tolerance must-pass slice
states what 150 cases can detect, with the standard error worked
says when the set re-runs and how it is refreshed

Where answers go wrong

"150 random transactions from last month." A quarter of a month's traffic holds about four own-account transfers, one tax payment and half a payroll run on average. The slice where a mistake reaches a filing cannot be measured, so the line cannot protect it.

"Have a model judge whether each category is reasonable." Each case has one right account and the signed ledger already holds it. A judge adds a call per case and an error rate of its own, and its "reasonable" admits exactly the near-miss drafts the accountant has to unpick.

"Ship if accuracy is at least 95% on all 150." One number cannot tell caution from error. Count queued cases as wrong and the line punishes a model for asking; count them as right and a model that queues everything passes with the operations manager doing all 600 by hand.