What changed and what did not
Separate the three parts of product work that models changed — the cost of a prototype, the shape of a spec, the meaning of done — from the parts they did not.
On this page
Discovery, prioritisation and outcomes are the same job they were before a model sat inside the feature. Three things changed, and each one changes a document or a decision rather than a philosophy: a working prototype now costs a day instead of a quarter, a spec now carries an eval and a cost line, and "done" is a pass rate on a set of examples rather than a green build. A PM who treats the whole job as new over-rotates on the model; one who treats nothing as new ships a feature whose acceptance criteria cannot be tested.
What did not change
The outcome still comes first. A feature is a bet that some metric moves — resolutions without a human touch, time to first reply, renewals — and the metric is chosen before the mechanism. The North Star lesson writes one as a product of inputs, and nothing in that arithmetic cares whether a model is involved.
Discovery still runs on interviews. The PM in this course talks to 3–5 customers a week, about 40 a quarter, about what happened last Tuesday. A model does not tell you which job a support lead is hiring a tool to do; the lead does, in the interviews, and the job turns out to be clearing a 130-ticket backlog by 6 pm, not "using AI".
Prioritisation is still arithmetic against a backlog. Reach times impact times confidence over effort gives the same 2,133 / 6,000 / 2,000 whether candidate C is an AI summary or a CSV export, and the boring candidate still wins the quarter. The AI candidate is more likely to sit at 50% confidence, which is a reason to buy information cheaply, not to change the formula.
What changed: a prototype costs a day
Before, finding out whether users wanted a feature meant building a version of it. A team of four engineers for a quarter is 12 person-months, or at about 21 working days a month roughly 250 person-days, spent before the first customer touched anything.
Now a forward-deployed engineer builds a working drafting prototype on the customer's first CSV export in a day, two at the outside. Two hundred and fifty days against one is a change of about 250× in the price of an answer, and that changes what the prototype is for: it becomes a discovery instrument, something you put in front of five customers in week one to move a confidence score from 50% to 80%, as the RICE lesson scores it. The one-day prototype lesson gives the protocol. The FDE course's version takes three days and ends in a sponsor demo, because it is built for one customer's deployment rather than for the product: the three-day prototype and the demo.
The trap is that a prototype that works is one commit away from a prototype that ships. A day's prototype tells you desirability — do agents use the drafts — and cannot tell you viability: the cost per request at 200,000 requests a day, the p95 under load, the failure rate on the tickets the demo never saw. Those come from the next two changes.
What changed: a spec carries an eval and a cost line
An acceptance criterion used to be a sentence a tester could mark true or false: the export downloads a CSV with these twelve columns. A model's output varies from call to call, so no sentence of that form describes it. The criterion becomes a pass rate on a golden set — 200 examples the team agrees are representative, each graded against a rubric — and the line in the PRD reads "92% on the golden set". It is the first thing this course's AI-feature PRD requires.
Two more lines join it, because the feature now has a per-request cost and a per-request latency that an ordinary feature did not. Assume $3 per million input tokens and $15 per million output. A prompt of 1,500 input tokens costs 1,500 × 3 ÷ 1,000,000 = $0.0045; a 300-token reply costs 300 × 15 ÷ 1,000,000 = $0.0045; so one request is about $0.009. At 200,000 requests a day that is $1,800 a day and $54,000 a month, a number the pricing lesson later has to clear against $60,000 of add-on revenue, and 1.8× the $30,000 the feature is budgeted. So the cost line carries two numbers: no request over $0.02, which catches a runaway thread, and a monthly mean under $0.005, the $30,000 spread over 200,000 × 30 = 6 million requests. The $0.009 clears the first and fails the second. The PRD's done line therefore reads: 92% on the golden set with no factual error above 0.5%; p95 under 1 s to the first token and under 4 s to the last; no request over $0.02 and a monthly mean under $0.005.
Latency is stated at the percentile because the tail is what users feel, and a 2 s median with a 6 s p95, which is what the drafting build measures, is a feature that is slow one time in twenty. The discipline is the one the system-design course insists on — numbers, not adjectives — applied to a document that used to get away without them.
What changed: "done" is a pass rate, and it can come undone
A green build proves that the code does what the code says, and it stays true until someone changes the code. A pass rate on a golden set is a statement about behaviour on examples, and it can change when nothing in the repository does: the vendor updates the model behind the API, and the same prompt against the same examples scores differently.
The mistake is to treat the eval as a launch gate and never run it again. Module 8 works through a case in which the pass rate falls from 92% to 86% the week after a model update — 12 more failures out of 200, concentrated in one ticket type — and the PM has to decide at what point to stop diagnosing and roll back. A team that ran the eval once at launch would find out from the customer.
So the eval becomes the feature's regression suite. It runs on every prompt change and every model update, and at the assumed rates it is cheap: 200 examples × $0.009 is about $1.80 a run, under four agent-minutes at an assumed $30 loaded hour. The evals lesson covers how the set is built and which failure classes block a launch regardless of the headline rate; the mechanics underneath — grading modes, judge bias — are in the developer's guide.
What this does to a week
The three changes land on the calendar more than on the org chart. Discovery still takes 3–5 interviews a week. Added to it: a prototype review most weeks, an eval run with its number, and a cost line in every spec review, since a feature can now be desirable, shippable and unaffordable at once. The FDE course shows the same week from the engineer's seat.
In an interview
The question comes as "how is PM work different with AI", and the interviewer is testing whether you can name the three mechanical changes rather than recite that everything is different. Name them with their numbers: a day instead of a quarter, a spec with a pass rate and a cost per request, a done line that a model update can undo. Then say what did not change, because a candidate who thinks discovery and prioritisation were replaced by prompting will ship a feature nobody asked for.
The mistake that loses points: calling a feature done on the demo. A demo is a prototype that worked on the examples the builder chose. The done line is a pass rate on examples the team chose, a p95 and a cost, and a candidate who cannot say what the three numbers are has not finished the feature.
Check yourself
1. A prototype built in a day is used by 4 of the 5 trial customers every day for two weeks. The engineering lead proposes shipping it as is. What does the day's prototype not tell you, and which number do you need before you agree?
It tells you desirability and nothing about viability. The missing number is cost per request at production volume: at the assumed $3 / $15 per million and 1,500 + 300 tokens, $0.009 a request is $54,000 a month at 200,000 requests a day, which the prototype never had to pay. Second is the pass rate on a golden set the team built, not the trial's own tickets. Agree when both are in the spec and both clear the done line. At $0.009 against the $0.005 mean the cost does not clear it yet, so the output cap and routing from tokens as a cost model come first.
2. The done line carries 92% on the golden set, p95 under 4 s to the last token, no request over $0.02 and a monthly mean under $0.005. The team proposes a larger model that lifts the pass rate to 95% and doubles the token price. Does it still meet the done line?
On cost, no. $0.009 doubles to about $0.018: a typical request is under the $0.02 ceiling by 10%, but 3.6× the $0.005 mean, about 200,000 × 30 × $0.018 = $108,000 a month against the $30,000 budget. On latency it already fails: the build measures 6 s at p95 against the 4 s line, and a larger model usually decodes more slowly, so the gap widens. Three points of pass rate buy back neither clause. Say both.
3. The eval ran once, at launch, and passed at 93%. Six weeks later a customer reports wrong refund amounts. What is the first thing to do, and what should have happened already?
Run the eval now and compare to the 93%; if it has moved, a model or prompt change is the first suspect and the diff is the diagnosis. What should have happened is a weekly run plus one on every prompt change and model update. At about $1.80 there was no cost reason not to, and it would have shown the drop within a week rather than six weeks later through a customer.