The one-day prototype
Use a prototype that costs one FDE day as the test at a leaf of the tree, read its three numbers, and say what it cannot tell you about cost or scale.
On this page
When a working prototype costs a day — one FDE, one model call, the tenant's last 100 tickets — it stops being the first step of delivery and becomes an instrument of discovery: the assumption test at the leaf of the tree. Its reading is desirability, measured in what three agents did with 60 drafts. It cannot read viability or cost, and a team that ships it has mistaken an instrument for a product.
What changed: a day, not a quarter
A prototype used to be a quarter of engineering. A team of four for three months is about 260 person-days at 22 working days a month, so nobody built one to test an assumption; they tested with mock-ups and interviews and built the real thing on faith. Now the prototype is a system prompt, twenty past tickets as examples, one model call and a text box beside the ticket, and an FDE builds it in a day. One of 260 person-days is 0.4% of the build, spent on the assumption most likely to kill it.
That ratio changes the order of work. The prototype comes before the scoping, before the PRD, before any estimate of effort, because for one day you can find out whether anyone wants the thing. What changed and what did not lists this as the first of the three changes; this lesson is the protocol for using it. The FDE's own version, built for one customer's deployment rather than for the product's tree, is the three-day prototype and the demo, and the two differ in purpose more than in code.
The protocol
The rule comes first, written before anything is built, because a rule written afterwards is a rationalisation. For the drafting branch: three agents, one afternoon, 60 tickets, and the branch is cut if fewer than 30 drafts are inserted into a reply. That is test T1 from opportunity solution trees, and it is the only reason the day is being spent.
The morning is the FDE's: the prompt, the examples, a side panel that shows the draft with an insert button. At noon the PM sits with the lead and the three agents and says what is being measured. The afternoon is 20 tickets each, with the PM timing every reply and logging one of three outcomes per draft: inserted unedited, inserted and edited, discarded. By evening there is a one-page snapshot, the same shape as an interview snapshot, carrying three numbers and the quotes.
The three numbers are the insert rate, the minutes per reply with a draft against the 3-minute baseline, and the count of drafts an agent called wrong. The first answers the rule. The second tests the minute that the whole job statement in jobs to be done rests on. The third is a first look at the error rate behind the 5% ceiling on wrong replies sent, and only a first look.
Reading the result
Say the afternoon returns 24 drafts inserted unedited, 14 inserted after an edit and 22 discarded. Inserted is 38 of 60, 63%, and the rule passes. Then the timing: replies with an inserted draft averaged 2.1 minutes, but the 22 discarded ones averaged 3.4, because the agent read the draft, rejected it and then wrote from scratch, which is slower than never seeing it. Blended: (38 × 2.1 + 22 × 3.4) ÷ 60 ≈ 2.58 minutes, a saving of about 0.42 minutes a reply rather than the 1 minute assumed. Across 560 tickets that is about 237 agent-minutes a day, roughly 92 replies at 2.6 minutes against the 130 in the backlog.
So the branch survives, and the job's arithmetic just got tighter. The next test is not more agents; it is the discard rate by ticket type. If 15 of the 22 discards came from refund tickets, the feature scoped to everything except refunds recovers most of the minute on the tickets it keeps, and refunds go back on the tree as their own opportunity. The prototype did not answer the question; it replaced a guess with a measurement and told you which measurement to take next.
What it tells you and what it cannot
It tells you desirability: whether agents inserted the draft, whether they came back to it after lunch, what they said while the PM was not asking. Three agents and 60 tickets are enough to reject a branch, which is what an assumption test is for.
It cannot tell you viability. Sixty requests say nothing about the bill for 200,000 a day; what the afternoon does give you is the shape of one request, roughly 1,500 tokens in and 300 out, and tokens as a cost model does the multiplication. It cannot tell you latency under load: 60 sequential calls on one afternoon are not a p95 across 12,000 seats. It cannot tell you the error rate: 60 tickets from one tenant, chosen by whoever was on shift, are not a 200-example golden set stratified by ticket type, and a draft the agent inserted is not a draft that was right. That rate comes from the eval in evals as the feature's test suite, and the prototype's 60 tickets are its first 60 candidates.
The honest sentence after a good afternoon is "three agents inserted 38 of 60 drafts and the reply fell from 3 minutes to about 2.6; we do not yet know the error rate, the cost or the latency at scale." Every clause of that sentence is a number or an absence of one.
The trap of shipping the prototype
The demo goes well, the lead asks when she can have it, and the prototype quietly becomes version one. It has no eval, no cost cap, no refused state, no handling for the ticket that is too long for the window, and one tenant's macros in its prompt. At the second tenant the macros are different and the drafts are wrong in ways nobody measured. The bill arrives at a scale the afternoon never touched. And the "done" line was never written, so nobody can say whether the feature is finished or merely on.
The move that feels wasteful is the right one: throw the code away and write the PRD with the eval set as the acceptance criteria, which is the AI-feature PRD. What survives the prototype is everything that mattered: the prompt's examples, the three numbers, the snapshot, and the 60 tickets as the start of the golden set. The code was the cheapest part of the day.
In an interview
What is being tested is whether you can say what a prototype proves and whether the decision rule existed before the build. Give the rule and the result as one sentence: "Three agents, 60 tickets, cut under 30 inserted; we got 38, and the reply fell to about 2.6 minutes rather than 2, so the branch lives and the next test is discards by ticket type." Then name what you still do not know, unprompted: error rate, cost, latency at load.
The mistake that loses points is reporting that the agents loved it without the insert count, or treating the 60-ticket afternoon as the eval. An interviewer who hears "it worked" will ask "on how many, and what was the rule", and a candidate without both answers has not run one.
Check yourself
1. The FDE can have the prototype ready in three hours, but only with one agent's macros loaded. Build it?
Yes, with the rule narrowed to match: that agent's ticket types only, and the numbers reported as a slice. A three-hour test on a slice today beats a three-day test that waits for all 200 macros, because the slice can still reject the branch. Write the slice on the snapshot so nobody reads 38 of 60 as a tenant-wide result.
2. Sixty drafts, 52 inserted, but the afternoon's tickets were the 20 easiest types. What do you know?
That the easy types are desirable: 87% insertion on that slice. Nothing about the other types, nothing about the error rate, nothing about cost. Next afternoon: 60 tickets sampled across all types in their real proportions, which may halve the rate, and that halved number is the one the PRD gets to use.
3. Sales wants the prototype in tomorrow's demo to a 3,000-agent prospect. Yes or no?
Demo yes, labelled as a prototype, on the prospect's own sample tickets, with the three numbers on the slide. Ship no: no eval, no cost line, no refused state. Say in the same breath what "done" would need — a pass rate on a golden set, a p95, a cost per request — and the date the PRD will carry them.