What "working" means: the eval set
Build an eval set from your own traffic, have the role's owner label it once, and write the acceptance line every prompt or model change must pass.
On this page
An automation that nobody has defined as working is working until a customer says otherwise. The eval set is the definition written down: 50 to 200 real cases from the automation's own traffic, each labelled once by the person who owns the role, scored against an acceptance line taken from the definition of done. It runs on every prompt or model change, so a change ships on a number rather than on someone's afternoon of reading outputs.
The mechanism — input, expected value and grader; the three grader kinds; what 200 items can detect — is taught in evals as the unit test of LLM work. This lesson is the owner's half: which cases, labelled by whom, held to what line.
It looked fine in week one
Support triage at the online retailer from approval gates, not the agency, is the running example for this module. It reads 3,000 tickets a day, puts each in one of 12 categories, auto-replies to the 30% it is allowed to answer, and queues the rest for a person. In its first week the founder read about 40 outputs, found two wrong, and called it done. In week six someone shortened the prompt to cut cost. Two weeks after that, the billing team noticed refund requests arriving in the how-to queue.
That is checking by reading. The 40 outputs came from one week's mix, the founder's verdicts were never written down, and the prompt change had nothing to be compared against. Even at face value, 38 of 40 is 95% with a standard error of √(0.95 × 0.05 ÷ 40) ≈ 3.4 points, so "95%" meant anywhere from about 88% to nearly 100%.
Cases come from your own traffic
A case is a real input the automation has seen, frozen with everything the model step received — the ticket text, the customer's membership, the last three messages — so it reproduces next month. Two draws fill the set:
- A random draw from the last 30 days, so the overall rate means something. For triage, 150 of the month's 90,000 tickets.
- A targeted draw for the inputs where a mistake costs most, because a random draw under-fills them. Membership cancellations are about 40 a day, 1.3% of tickets, so a random 150 holds two. The targeted 50 fill the must-pass slice and the three exception classes from definition of done and the exception list: unknown input (a ticket in Portuguese), ambiguous rule (a refund request that is also a bug report), external failure (an order lookup that returned nothing).
Size follows volume and the cost of an error:
| Automation | Volume | Cases | Why that many |
|---|---|---|---|
| Support triage | 3,000 a day | 200 | 12 categories and a must-pass slice; at 200 the standard error is under 2 points |
| Bookkeeping prep | 600 a month | 100–150 | a dozen accounts carry most of the volume; the signed ledger already holds the labels |
| Inbound lead handling | 400 a month | 50–80 | three sources and two outcomes; a month of traffic is barely larger than the set |
Synthetic cases written by a model are no substitute: they come in the model's own style and never contain the odd field your connector delivers.
Labelled once, by the owner
The label is the owner's answer, because the owner is the person whose definition of done is being encoded — not the vendor and not whoever built the workflow. At about a minute a ticket, 200 cases take 3.3 hours; call it 4 with a second look at the cases where the model and the owner disagree. At the course's assumed founder rate of $100 an hour that is $400, once. Delegated to a support lead at the assumed staff rate of $30, it is $120, with the founder settling the twenty or so disputed cases.
The labels are frozen, versioned beside the prompt, and changed only at a review with a written reason — usually because the policy changed, never because the model disagreed. A set that gets relabelled whenever an output looks plausible measures nothing.
Running the set is cheap by comparison. A triage case is 2,200 input tokens and 250 output; at the course's assumed $2 per million input tokens and $8 per million output, 200 cases cost 440k × $2/M + 50k × $8/M = $0.88 + $0.40, about $1.30 a run.
The acceptance line comes from "done"
Triage's definition of done: every ticket carries one of the 12 categories, and either an approved auto-reply or a place in a human queue within 15 minutes. The acceptance line turns that sentence into checks a run passes or fails:
- category correct on at least 188 of 200 (94%), graded by exact match against the owner's label; today's prompt scores 192, the 4% mis-route rate that sampling review measured
- zero auto-replies on the must-pass slice — the 20 refund-dispute, cancellation and legal tickets that must always reach a person
- every case ends in a terminal state — categorised and answered, or queued; none left in between
A rate with a floor and a list with zero tolerance are different instruments. The rate tolerates a few mistakes, because a mis-filed how-to ticket costs an agent three minutes to re-route. The must-pass list tolerates none, because one auto-reply to a refund dispute can cost the customer.
How large a move 200 cases can see decides how the floor is read. At 96%, the standard error is √(0.96 × 0.04 ÷ 200) ≈ 1.4 points, so 192 against 189 is noise and 192 against 182 is not. Comparing two runs case by case — which cases flipped, and in which category — is the method in regression gates. The owner's job is to write the line down before the first run, so it cannot drift to meet the result.
Run it on every change
A change is anything that can move a case: the prompt, the model or its version, a tool description, the connector's field mapping, the list of categories. Each runs the set before reaching live traffic, and the score, the must-pass verdict and the cost per ticket are recorded beside the change.
One change can arrive without a commit: the provider moves an alias to a newer model, or changes the serving stack under a pinned one. So the set also runs on a schedule, and a drop with nothing merged points at the provider. Catching failures between runs is the job of drift and silent failure.
The set ages as the traffic changes. Once a month, add ten cases from the reviewer's corrections — the tickets the sampled review found wrong — and retire cases for categories that no longer exist.
In practice
When a vendor or builder says the automation works, ask: "What does it score on our 200, against the line we wrote, and what did the last change do to it?" A vendor's benchmark is someone else's traffic; a demo is inputs the builder chose. Tell the team the acceptance line before the first run, and that the set runs on every change, theirs included.
The mistake that costs money is accepting "it works" from a demo instead of a score on your own labelled traffic. Its bill arrives weeks later as refunds in the wrong queue, with no record of when it started.
Check yourself
1. Bookkeeping prep categorises 600 bank transactions a month, and the external accountant has signed three months of ledgers. How many cases, and who labels them?
About 150: 120 drawn from the 1,800 signed transactions and 30 targeted at the rare, costly ones — transfers between your own accounts, tax payments, refunds. The signed ledger is already the owner's label, so labelling costs only the 30 targeted cases, about an hour of the founder's time at $100. Check first that the ledger uses the same accounts the automation will be held to.
2. A prompt change moves triage from 192 to 189 of 200, every must-pass case still passes, and cost per ticket falls 20%. Ship?
Probably. Three cases is 1.5 points, about one standard error of 1.4, 189 is above the 188 floor, and the must-pass list held. Read the case-by-case diff first: if all three losses sit in one category, hold and look at that category; if they are scattered and a repeat run lands in the same place, ship and take the 20%.
3. A vendor offers to build the eval set from 1,000 synthetic tickets their model writes. Accept?
No. The cases would be in the model's style, labelled by nobody who owns the role, and missing the inputs your connector really produces. Take 200 from your own traffic and spend the four hours labelling them; synthetic tickets can be practice cases, never the set that gates a change.