Writing · The AI-feature PRD

The PRD with no numbers

Add the eval, latency, cost and failure-mode requirements a thread-summary PRD is missing, and derive each number from the feature's own budget and break-even.

critiquemediumExercise · 3 min · 44 of 59

On this page

A PRD for ticket-thread summaries reaches review with every requirement written as an adjective. The AI-feature PRD says what four of them have to become; this one has none, and the numbers it needs are all in the room.

The situation

The PRD goes to review on Thursday. Its requirements section, in full: "Summaries must be accurate and concise. The panel must load fast. Cost should be reasonable. Done when the pilot support leads are happy with summary quality." The summary appears when an agent picks up a handed-over ticket, about 30,000 times a day across tenants, 900,000 a month, and the feature's budget is $10,000 a month. A thread averages 2,000 words; the longest 5% run past 6,000, and nothing caps them. The system prompt is 400 tokens and a summary about 100. Assume 1.3 tokens a word, $3 per million input tokens and $15 per million output. Reading a long thread takes an agent about 4 minutes and a summary about 1; assume a summary that misstates the thread costs a 20-minute escalation, as a wrong reply does in the drafting feature. In the prototype, first tokens arrived at p95 0.8 s, and agents read the summary as it streamed.

The question

Add the four requirements this PRD is missing, each with a number, and say where each number comes from.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Missing: an acceptance criterion the team can run, a latency, a cost, and a decision per failure mode.

Acceptance. 90% on a 200-thread golden set labelled by the pilot leads, with any summary that drops a commitment to the customer (a refund, a callback date) blocking at more than 1 in 200. The 90% is break-even with margin: a summary saves 3 minutes and a wrong one costs 20, so past 3 ÷ 20 = 15% wrong it costs time. At 90% on 200 cases the margin is about ±4 points, which keeps the measured rate just clear of 85%. "The leads are happy" becomes the leads labelling the set.

Latency. First token under 1 s at p95. Agents read as it streams, and 75 words take longer to read than to generate, so nobody waits for the last token; 1 s is where attention leaves the panel. The prototype's 0.8 s shows it is reachable.

Cost. A mean under $0.011 a summary, $10,000 ÷ 900,000. The average thread is 2,000 × 1.3 + 400 = 3,000 tokens in, $0.009, plus 100 out, $0.0015: $0.0105, inside by 5%. A ceiling too: cap input at 8,000 tokens, oldest messages dropped, and output at 150, so no summary costs over about $0.026. Uncapped, a 20,000-word thread costs $0.08.

Failure modes. Wrong: one click marks it, and the thread joins the golden set. Refused or empty: no panel, the thread as today. Slow past 1 s: a skeleton above the thread, which stays readable. Over budget: the input cap, not a switch-off.

What would move these: a measured escalation cost far from 20 minutes moves the 90%.

Rubric

Replaces "the leads are happy" with a pass rate on a named golden set and a blocking class.
Derives the threshold from 3 minutes saved against a 20-minute escalation, and checks the set's margin against the 85% break-even.
Chooses first-token latency for a summary read as it streams, and says why the last token is not the number.
Computes the mean cost per summary from tokens and compares it with the budget's $0.011.
Adds a per-summary ceiling enforced by an input and an output cap, with the uncapped long-thread cost shown.
Gives wrong, refused, slow and over budget a decision each.

Where answers go wrong

"Copy the drafting PRD: 92%, p95 under 4 s, under $0.02 a request." Numbers with another feature's sources. A summary breaks even at 85%, not where a draft does; last-token p95 is the wrong end point for text read as it streams; and a $0.02 ceiling says nothing about this budget's $0.011 mean.

"Leads rate summaries 4 out of 5 in the pilot." A satisfaction score is not an acceptance criterion: it cannot be run before launch, cannot block a build, and moves with the mood of five people.

"Cost: under $10,000 a month." True and unusable at review. Nobody can check a monthly total against one call, and with no per-call ceiling one tenant whose threads run long moves the mean, unseen until the bill.