Writing · The AI-feature PRD

Write the done line

Write the eval, latency and cost clauses that define done for chat drafting, defend each against a looser line the build meets, and decide what launches.

casehardExercise · 4 min · 45 of 59

On this page

The engineering lead has written a done line the build already passes. The job is to write the one the feature needs, using the method in the AI-feature PRD, and then say what this build is allowed to launch.

The situation

The drafting feature is moving to live chat, where the customer is waiting and each agent runs three chats at once. The forecast is 50,000 chat drafts a day across tenants, 1.5 million a month, and the budget is $9,000 a month. Assume $3 per million input tokens and $15 per million output. Drafts are sent whole, and the screen shows its slow state at 4 s, as in the email PRD. The 200-case chat golden set, labelled by two support leads, mirrors traffic: 140 short chats score 134 (95.7%), 60 policy chats score 50 (83.3%), 184 overall (92%); two drafts quote a wrong price, both policy. Latency on 2,000 replayed chats: first token p95 0.9 s; last token p50 1.6 s overall, and p95 2.2 s on short chats, 6.8 s on policy, 5.9 s blended. A request averages 1,400 tokens in and 100 out, and the longest 1% of prompts run to 9,000 tokens. The engineering lead proposes "p50 under 2 s, 90% on the golden set, average under $0.01 a request".

The question

Write the done line for chat drafts, defend each number against the engineering lead's, and say what this build may launch.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Done: 92% on each class of the chat golden set, no factual error above 0.5%; p95 under 1 s to the first token and 4 s to the last; no request over $0.015, monthly mean under $0.006.

Eval. 92% is the drafting threshold, set by the 20-minute escalation, applied per class because a blend hides the weak one: 92% overall is 95.7% and 83.3%. Sixty cases are good to only about ±10 points. Two wrong prices in 200 is 1%, twice the blocking rate.

Latency. Last token, because a chat draft is sent whole; 4 s, where the screen hands control back; p95, because the median hides the one chat in twenty a waiting customer remembers.

Cost. $0.006 is $9,000 ÷ 1.5 million; the mean is (1,400 × $3 + 100 × $15) ÷ 1,000,000 = $0.0057, inside. The ceiling comes from capping the prompt at 3,000 tokens and output at 300: $0.009 + $0.0045 = $0.0135, where a 9,000-token prompt costs about $0.03.

What launches. The mean passes; the caps go in first. Short chats pass the rest: 95.7%, no wrong price, 0.9 s and 2.2 s. Policy chats fail eval, facts and latency. Launch short chats only, 70% of volume; policy chats keep macros until a re-run passes on a slice grown to 200.

The engineering line passes by testing nothing that fails: p50 ignores the 6.8 s tail, 90% has no origin and lets both wrong prices through, and $0.01 average allows $15,000 a month against $9,000.

Reverse it if a retrieval fix takes policy chats to 92% with no wrong price and p95 under 4 s, or if agents send under half the short-chat drafts.

Rubric

Writes three clauses, eval, latency and cost, each a number with a percentile, a set or a budget behind it.
Applies the threshold per class, shows the blend hiding the 83.3% policy slice, and blocks on the two wrong prices.
Picks last-token p95 for a draft sent whole and ties 4 s to the slow state rather than to the build's measurements.
Derives the mean from the budget and the ceiling from token caps, with the uncapped cost shown.
Turns the result into a scope decision, short chats now and policy chats later, rather than a whole-feature yes or no.
Says why the engineering line passes and names the evidence that would widen or shrink the launch.

Where answers go wrong

"Accept the engineering line; the build meets it." A done line written after the measurement describes the build. Its p50 never sees the 6.8 s tail, and its 90% has no origin and lets two wrong prices reach customers who are waiting in the chat.

"92% blended and p95 under 6 s, so everything launches." This moves the line to where the build is. The 6 s is the launch guardrail, not the requirement, and a blended 92% averages a class that is wrong one time in six into a passing number.

"Hold the whole launch until policy chats pass." It throws away the 70% of volume that meets every clause today, and with it the short-chat data that would say whether the slice is worth having.