The AI-feature PRD
Write a PRD whose acceptance criteria are an eval, whose latency and cost per request are numbered requirements, and whose failure modes each have a decision.
On this page
An AI-feature PRD is an ordinary PRD with three sections rewritten as numbers. The acceptance criteria become an eval, a pass rate on a golden set built before anyone chose a model; latency and cost per request become requirements with numbers; and each way the model can fail — wrong, refused, slow, over budget — gets a product decision written before launch. For the drafting feature the done line reads "92% on the golden set, p95 under 4 s, under $0.02 a request", and every clause has a source a reviewer can check.
A criterion nobody can run
The first draft's acceptance criterion was the ordinary kind: "Given a ticket, when the agent clicks Draft, a helpful and accurate reply appears." It is true on the demo, false on some fraction of tickets, and silent on which fraction, so no tester can mark it. The rewrite is a test the team can run, on the set evals as the feature's test suite builds: 92% of the 200 cases pass the four-class rubric, with no factual error above 0.5%, on the shipped prompt and model versions, re-run on every change.
The eval clause
The 92% comes from the job. In jobs to be done a wrong reply that is sent costs a 20-minute escalation, so more than 27 wrong in 560, 4.8%, eats the whole saving; agents read every draft, which turns 8% wrong drafted into well under 4.8% wrong sent. The 200 comes from the margin: 92% measured on 200 cases is good to about ±3.8 points, enough to tell 92% from 85%. The 0.5% comes from the contract clause on factual errors: a build at 94% with two wrong prices is blocked, and one at 92% with none ships.
The clause has a price. 8% of 200,000 drafts is 16,000 a day for agents to fix or discard, about 89 agent-hours at the 20 seconds each that model selection uses, against 1,300 to 3,300 hours returned by 0.4 to 1 minute saved per reply. With auto-send the same 92% is 16,000 wrong replies a day to customers, so the clause is valid only beside the out-of-scope line: never sent without the agent.
The latency clause: a percentile and an end point
First token, p95 under 1 s: when the agent can start reading, and 1 s is Nielsen's limit for an unbroken train of thought. Last token, p95 under 4 s: when the agent can insert the draft, since a reply is sent whole.
The p95, because at 40 tickets a day the slowest 5% is two drafts per agent per day, which a median ignores; tail latency is the general case. The 4 s, because it is where the screen hands control back: past it the composer shows cancel and the agent can type over the draft, as the non-deterministic UI designs it. The evidence that would move it is behavioural. Logs showing agents typing over drafts at 3 s make the line 3.
The build as specified fails — reading an architecture measured 6 s at p95 — and the PRD says so and names the path. The output cap halves decode time on the 70% of drafts that are short, and the small tier they route to decodes faster again, so the two cost levers are latency levers too; the p95 is re-measured after both, on replayed traffic. If it is still over 4 s, the honest options are to launch the slice that meets it or to hold. Editing the 4 to a 6 is not one of them.
The 6 s in AARRR and guardrails is a different line: it pauses a ramp already under way.
The cost clause: a ceiling and a mean
As specified, a request is 1,500 input tokens at an assumed $3 per million plus 300 output at an assumed $15 per million: $0.0045 + $0.0045 = $0.009, comfortably under $0.02. The month is $54,000, 1.8× the $30,000 budget, as tokens as a cost model works out. One per-request number passed a feature that loses money, so the clause carries two.
The ceiling: no request over $0.02. Two caps enforce it on every call — the prompt at 4,000 tokens with history trimmed oldest first, the output at 500 — so the most a call can cost is (4,000 × $3 + 500 × $15) ÷ 1,000,000 = $0.012 + $0.0075 = $0.0195. It catches the runaway thread or retry loop. The mean: under $0.005 a request, the $30,000 budget ÷ (200,000 × 30) = 6 million requests a month. As specified the mean is $0.009 and fails; after the output cap and routing the bill is about $27,500, $0.0046 a request, inside by 8%.
Four failure modes, four decisions
Each mode gets a trigger with a number and a decision; the screens are in designing for "the model might be wrong".
- Wrong: 16,000 drafts a day at 92%. Never sent without the agent; insert, edit, regenerate and report are one action each, and each report is a candidate golden-set case. A fact-class rate over 0.5% on a weekly run rolls the version back.
- Refused: the model declines. No retry, since the same prompt usually refuses again: an empty composer with a one-line reason. A vendor error is different: retried once, silently, with the same idempotency key.
- Slow: no last token by 4 s. Cancel appears and the composer stays editable; if the agent starts typing, the draft collapses rather than landing on their text. At 10 s the stream stops, with no automatic retry.
- Over budget: an over-cap request is trimmed, never the system prompt or the ticket. A tenant at its contract's monthly cap sees the reason and the reset date, never an error, with its admin warned at 80%; a tenant with no cap running 20% over its share pages the PM and the account owner.
The done line, in full
Done: 92% on the 200-case golden set with no factual error above 0.5%; p95 under 1 s to the first token and under 4 s to the last; no request over $0.02 and a monthly mean under $0.005; measured on the shipped prompt and model versions, and re-measured on every change.
Read against the build as specified, it passes twice, on the eval and on the 0.9 s first token from the non-deterministic UI, and fails three times: the PRD doing its job before the build exists. The FDE writes the same three lines for one customer in definition of done for an AI deployment, and adds coverage and a named owner to make five.
In an interview
The prompt is "write the PRD for an AI feature" or "how would you know it is done". The interviewer is testing whether your done line has three numbers with a source each: "92% on a 200-case golden set, because a wrong reply sent costs a 20-minute escalation; p95 under 4 s to the last token, because that is where the screen hands control back; no request over $0.02 and a mean under $0.005, because the ceiling alone passed a feature 1.8× over budget."
The mistake that loses points is a done line written from the build's own numbers — "p95 under 6 s, because that is what we measured" — which describes the build rather than the need and passes whatever ships. The second is "users are happy" as the acceptance criterion: it cannot be run before launch and cannot block a build.
Check yourself
1. After both levers the p95 to the last token measures 4.6 s. Engineering proposes changing the requirement to 5 s. Decide.
No. The 4 s comes from the screen, not the build: at 4.6 s more than one draft in twenty reaches the cancel state, more than two a day per agent. Measure the short-reply slice on its own; if it meets 4 s, launch that slice and hold the long replies. The line moves only on evidence about agents — logs showing they do not type over a draft until 5 s — never on evidence about the build.
2. A tenant with 1,200 agents asks for auto-send on password-reset tickets, about 4,800 a day. Their slice of the golden set scores 39 of 40. Do you change the wrong row?
Not on 40 cases. At 97.5% on 40 the margin is 2 × √(0.975 × 0.025 ÷ 40) ≈ ±4.9 points, so the slice cannot tell 97.5% from 93%, and 93% is about 340 wrong replies a day sent to customers. Label at least 600 cases of the slice, the fewest at which zero factual errors bounds the rate under 0.5% by the rule of three, and measure an undo rate; the row changes only in a PRD revision backed by both.
3. Sales signs a 1,000-agent tenant whose tickets are long email chains: prompts average 3,000 tokens, on the mid tier. Does the cost clause catch it?
The ceiling does not: (3,000 × $3 + 300 × $15) ÷ 1,000,000 = $0.0135, under $0.02. The mean does: 2.7× the $0.005 line, and at 1,000 agents × 40 tickets, $540 a day, about $16,200 a month — over half the budget for one tenant. Price it per use or lower their history cap before signing.