Evals as the feature's test suite
Build the 200-example golden set, rubric, pass threshold and blocking failure classes that the PRD's done line points at, and re-run it when the model drifts.
On this page
The PRD's done line for the drafting feature reads "92% on the golden set, p95 under 4 s, no request over $0.02 and a monthly mean under $0.005", and the first of those is a test suite the PM owns. A golden set of 200 labelled tickets, a rubric that scores each draft, a pass threshold, and a short list of failure classes that block launch at any rate: that is what "done" points at, and the contract between the PM, the FDE and research in working with an FDE and with research.
The golden set
Two hundred real tickets, drawn so that each ticket class is present in proportion — 140 short replies, 40 policy questions, 20 escalations, the 70/20/10 mix the router in model selection relies on — with a reply the support lead wrote or approved before anyone chose a model. The labelling is the expensive part: at ten minutes a ticket, 200 tickets is about 33 hours of a senior agent's time, roughly $1,000 at an assumed $30 an hour loaded. It is the best $1,000 the feature spends, because every later decision is measured against it.
Why 200 and not 50. A pass rate is a proportion, and a proportion measured on n examples has a margin of about ±2 × √(p(1−p)/n). At p = 0.92 and n = 200 that is ±3.8 points; at n = 50 it is ±7.7 points, wide enough that 92% and 85% are the same number. Two hundred gets the margin under 4 points for about $1,000; a thousand gets it under 2 points for $5,000, which is worth paying once the feature is live and the question is a 1-point drift. Building the eval set in the FDE course does the same job for one customer's deployment.
The rubric
Each draft is scored against its reference reply on a rubric, not on exact text, since two good replies differ in wording. The rubric for the drafting feature has four failure classes, and a draft fails if it fails any:
- Wrong on a fact — a price, a policy, a date, an article the draft cites that does not say what the draft says.
- Wrong action — promises a refund the policy does not allow, or escalates a ticket that has a stock answer.
- Tone — off the house style: apologises in the first line, over-long, missing the sign-off.
- Form — not a reply at all: refused, empty, truncated, or not the structured shape the composer expects.
Who scores. Exact-match grading works for the form class (did the output parse). The fact and action classes are scored by a person for the first few runs, then by a second model reading the rubric, checked against the person's scores until they agree on 95% of cases. Tone is the hardest to grade and the cheapest to be wrong about. Exact, rubric and model-graded is the mechanics; the PM's job is to own the rubric text and the agreement number.
The threshold and what blocks
Ninety-two percent is the pass threshold — 184 of 200 drafts acceptable. It comes from the discovery arithmetic: in jobs to be done a wrong reply costs a 20-minute escalation, and more than 27 wrong a day eats the saving on 560 replies; 27 ÷ 560 is 4.8%, so the feature pays for itself below about 5% wrong sent, and 8% wrong drafted with agents catching most of them lands under that.
But the classes are not equal, and the threshold is not the only gate. A factual error in a customer-facing reply has a contract clause attached; the launch rule is that the fact class blocks at any rate above 0.5%, one draft in 200, regardless of the headline number. A build scoring 94% with two factual errors is blocked; a build scoring 92% with one ships. Tone failures do not block; they go on the backlog. Writing the blocking classes down is the difference between an eval that gates and an eval that decorates.
Drift after a model update
The eval runs on every prompt change and on every model update, which is where teams get caught. The vendor announces an update as an improvement; it is also a change to every draft, and only a re-run of the 200 says what changed. The execution interview exercise is exactly this case: pass rate fell from 92% to 86% the week after an update, and segmenting by ticket class, tenant and prompt version says whether to re-prompt or roll back to the previous model version, which the vendor keeps available for a window.
The eval costs almost nothing to run: 200 tickets × 1,800 tokens is 360,000 tokens, under $2 at the assumed prices and about ten minutes. Run it weekly as well as on every change, and plot the pass rate per class. A line that drifts down over a month with no change on your side is a vendor change you were not told about.
The set grows from the field
Every field failure — a draft an agent reported, an escalation traced to a wrong reply — becomes a new golden-set case with the right answer attached. The FDE's field notes from failure modes on real data are the main source. The set grows past 200 and the margin tightens with it. The one rule: a case is never removed or edited because the model fails it. An eval that is edited to pass is a demo with a spreadsheet.
The done line
Three numbers in the AI-feature PRD: the eval score, the p95, the cost per request. The first is defined here: "92% on the 200-case golden set, with no factual error above 0.5%, on the shipped prompt version and model version, re-run on each change." Anyone with access to the run can check that sentence, which makes it a definition of done rather than a hope.
In an interview
What is being tested is whether you can say what "done" means for a feature whose output is not deterministic, and defend the number. Give the four parts (set size with the margin it buys, rubric with its classes, threshold with where it came from, blocking classes with the rate) and say when the eval re-runs. "200 cases is ±4 points; 92% comes from the escalation cost; the fact class blocks at 0.5% because of the contract clause; it re-runs on every model update" is a complete answer in four sentences.
The mistake that loses points is a threshold with no origin — "92% because it sounds high" tells the interviewer the number is decoration. The second is a single pass rate with no blocking classes, which lets a factual error ship inside a good headline. The third is an eval that was run once, before launch, and never again; the interviewer will ask what happened after the next model update, and the honest answer is "we did not know".
Check yourself
1. The new build scores 93% overall; three of the 200 drafts have a wrong price. Does it ship?
No. Three factual errors in 200 is 1.5%, three times the 0.5% blocking rate, and on 200,000 drafts a day that is about 3,000 wrong prices a day for agents to catch. The 93% headline is irrelevant to the gate. Find the three, check whether the retrieved chunks were stale or the model ignored them, fix that, re-run.
2. The team wants to cut the golden set to 60 cases so the eval runs faster. The eval costs under $2 and ten minutes at 200. What do you say?
The margin at 60 cases is about ±7 points, so the eval could no longer tell 92% from 85%, and the threshold stops meaning anything; the time saved is seven minutes and the money about $1.30. Keep 200. If speed matters, run the 60 most recent cases hourly as a smoke test and the full 200 on every change and weekly.
3. A tenant with 1,200 agents reports that drafts "got worse" this week. The weekly eval still shows 92%. What do you check first?
Segment the run by tenant: the 200-case set, drawn across all tenants, holds about 20 from this one, its tenth of the seats: a ±12-point margin, too wide to see anything. Pull 50 of this tenant's tickets from the week, label them, run them. If that slice fails, it is a per-tenant cause — their help centre changed, their index is stale, their prompt version differs. If it passes, go to the field reports and look for a new ticket type the set does not cover yet, and add it.