Building the golden set
Source, label and stratify a 200-item golden set in an afternoon of an expert's time, keep it free of leakage, and say what each slice of it can detect.
On this page
The golden set is the list of cases the eval runs, and every number the eval reports is a statement about those cases, not about your traffic. A set that resembles the traffic makes the pass rate mean something; a set of the tickets the engineer remembered makes it mean "the prompt handles the tickets the engineer remembered". Building one costs a few hours of a domain expert's time, and the decisions in those hours — where items come from, who labels them, how many each slice gets, which items the prompt author may see — matter more than the choice of grader.
Three sources
Transcripts. Real inputs from production logs, or from the queue the feature will replace: real phrasing, real length, real mess — forwarded threads, three questions in one message, a pasted error. The distribution is the traffic's, which is the point and the limit: rare cases are rare here too, and nothing in last month's logs covers the policy that launches next month. Personal data is scrubbed before a transcript enters the repository.
Hand-written. Cases a person writes on purpose: the boundary between two labels, the policy edge, the injection attempt, the ticket behind last quarter's incident. They cover what matters most and appears least. Their weakness is the author. An engineer's invented ticket reads like an engineer wrote it — complete sentences, one question, the keyword present — and the model finds it easier than the real thing.
Synthetic. Variations a model generates from a seed: twenty rewordings of a rare cancellation request, one ticket in four languages. Cheap volume for a thin slice, with three risks: the generator's style is uniform, the generator may be the model under test and find its own phrasing easy, and a variation of a set item is that item reworded. A person reviews synthetic items before they enter the set, and they stay a minority — a fifth at most — of any slice whose number you report.
The mistake to recognise: 500 tickets generated by the model that classifies them, labelled by the instruction that generated them ("write a billing ticket"), scoring 98%. That set measured whether the model agrees with itself.
Labelling is the cost
Assume about 60 seconds to read a ticket and assign a label. Two hundred items is 200 × 60 s = 12,000 s, about 3.3 hours. Then a second pass over the disagreements: run the current prompt on the set and look again at every item where model and label differ, about 20 if the prompt is near 90%. Those are the hard items, two or three minutes each, about 50 minutes, so call the whole job 4 hours. Separately, a second person labels a 30-item sample independently, half an hour of their time.
The labeller is the person who owns the decision the feature automates — the support lead, not the engineer. The engineer knows what the prompt says; the lead knows what the right queue is, and the labels are the definition of correct for every run that follows.
The second labeller's sample sets a ceiling. If the lead and a senior agent agree on 27 of 30, people agree on about 90% of these items, and a model at 91% on them is at the ceiling, not below it. Items the two disagree on get a written resolution — the rule that decides them, added to the labelling guide and usually to the prompt — or go to a known-ambiguous slice that nobody gates on.
Slices, and why random sampling starves them
One pass rate over 200 items hides where the errors are. A slice is a subset whose failures cost something different or whose inputs behave differently: a label, a language, the boundary cases, the injection attempts. Each reported slice needs at least 20 items, and the arithmetic says why that is a floor rather than a target. At a 90% pass rate the standard error on 20 items is √(0.9 × 0.1 ÷ 20) ≈ 6.7 points, and one flipped item is 5 points. Twenty items detect a slice collapsing — 15 points or more — which is what a slice check is for. They do not detect a 3-point drift.
Twenty is more than a random sample gives a rare slice. Cancellations are 3% of traffic, so a random 200 holds about six. Stratify instead: choose the slices first, fill each to its floor, and let routine traffic take the rest.
Oversampling changes what the overall number means. Report two: the per-slice pass rates, and an overall rate weighted by each slice's share of traffic, which estimates what users see. The unweighted total over a stratified set describes nothing; leave it off the report.
Splits and leakage
The prompt author reads failures and writes rules for them — the loop prompt design as interface design teaches. Do that on the items you report and the set stops measuring anything: a tie-break rule written for item 142 fixes item 142, and the score rises by the number of items the author has read, not by what the rule does on traffic.
So keep two sets. A dev set of 50 to 100 items, which the author reads freely, mines for few-shot examples and writes rules against. A held-out golden set of 200, whose individual failures are opened only after the numbers have decided. Few-shot examples come from dev only, and the harness enforces it: hash each example and each golden input, and fail the run on a match. Six examples copied from the set score those six items at 100%: up to three points of 200 gained by leakage.
Leakage also comes in sideways. The same customer's thread, forwarded twice, lands in both sets: deduplicate on normalised text before splitting. A synthetic variation of a golden item is the golden item: generate only from dev seeds. A time split helps — golden from the latest month, dev from earlier — so the prompt is always scored on traffic its author has not read.
The set grows from the traffic
A set built once drifts away from the traffic it describes: a product launches, a new failure appears, a slice grows. Two rules keep it current. Every production failure someone has to fix becomes a case, the way a bug becomes a regression test. And each week a sample of traces — drawn mostly from tasks that failed a check or ended in a cap, plus a few random ones — is labelled, and the new kinds of failure are added, as observability and tracing describes.
Version the set, and after adding cases re-run the current prompt on the new version before comparing: two versions' pass rates measure different things. The sampling lesson adds one more use for the set's disagreements: they are where the 20-run repeat test belongs.
In an interview
The question "how would you build the eval set?" tests whether you know the set is the expensive, decisive part. Say the sources and what each is for; the labelling arithmetic with a named labeller — "200 items at a minute each is 3.3 hours of the support lead's time, call it 4 with a second pass"; the slices with a 20-item floor and a traffic-weighted overall; the dev and held-out split, with few-shot examples only from dev.
The mistake that loses points is "we'd generate a thousand synthetic examples with the model" — a set with no owner's labels, in the model's own style, often from the model under test. Second: a random 200 that holds four items of the slice where failures cost the most.
Check yourself
1. Contracts take 90 seconds to label, and you want 500 items. How long, and whose time?
500 × 90 s = 45,000 s, 12.5 hours for the first pass. A second pass over the 15% that disagree, at about three minutes each, is 75 × 3 = 225 minutes, nearly 4 hours. About 16 hours of the contract owner's time, two working days — a scheduling decision to make before the build starts, not after.
2. Refunds go wrong in a slice that is 2% of traffic, and your random 200-item set holds four of them. What do you add, and what do you report?
Fill the slice to at least 20, better 30, from transcripts first and hand-written cases second: 26 more items, about half an hour of labelling. Report the slice's own pass rate and an overall weighted by traffic share, so the extra refund items neither inflate nor sink the headline.
3. The pass rate jumped from 88% to 96% after a revision added eight few-shot examples, five of them copied from the golden set. What do you report?
Not 96%. The five copies can account for up to 2.5 points by leakage alone. Move those five items to dev, re-run, and report that number; if it is still well above 88%, the examples helped. Then add the hash check, so the next copy fails the run instead of flattering it.