Working with an FDE and with research
Run the weekly loop between field failures, the eval set and the capability boundary, and make the one decision a week that keeps all three honest.
On this page
Three roles touch a model-backed feature and each sees a different part of it. The FDE sees it fail on a real customer's tickets; research knows what the model class can and cannot do; the PM owns the eval set, which is the one artefact all three can read. A week in which the field report, the eval run and the decision happen in that order, once each, is a week in which the feature got better. A week in which they happen in any other order, or not at all, is a week of opinions.
What each role brings
The FDE feeds back field failures. Not "the drafts are sometimes wrong" but a one-page field note in the form the FDE course teaches: the failure, the data behind it and what the FDE thinks generalises. A good one reads like this. The tenant's 1,200 agents produced 48,000 drafts a day last week and edited 14% before sending. Agents flagged 60 drafts with the report button; 35 of the 60 were about refunds, 15 about tone, 10 miscellaneous. Of the 35, most quoted a 14-day refund window where the tenant's policy is 30 days for annual plans.
Research owns the capability boundary: what the current model class does reliably, what it does with the right context, and what it does not do at any prompt. The PM's question to research is never "can the model do this" in the abstract. It is "here are ten cases it fails; is that a boundary or a context problem", and the answer decides whether the fix is a prompt change the FDE ships in a week or a product decision to route the case to a human.
The PM owns the eval set, and the eval set is the contract. The FDE's field note becomes cases in it; research's boundary call is made against it; the PM's decision is a change to its pass rate or to its scope. The evals lesson covers how the 200 examples are chosen and graded; this one is about what moves through it each week.
The week, with numbers
Monday: one field report. The FDE's note arrives with the 35-of-60 refund figure. The PM reads it as a sample first: 60 flags out of 48,000 × 5 = 240,000 drafts in the week is a 0.025% report rate, and agents report what annoys them, not what is wrong. The 14% edit rate is the better signal of error; the 60 reports say which class of error annoys agents most.
The PM writes 10 refund cases for the golden set, drawn from the flagged tickets with the tenant's policy text attached as the expected source. The set goes from 200 to 210 cases.
Tuesday: one eval run. Before the new cases, 92% of 200 is 184 passes. The 10 refund cases pass 2, so 186 of 210 = 88.6%. The headline rate fell 3.4 points, and the fall is entirely the new cases — the original 184 still pass. The class matters more than the headline: all 8 new failures are wrong on a fact, a refund window or amount, and with the one factual failure the original 200 already carried, the fact class is 9 of 210 = 4.3%, past the 0.5% blocking rate, so the feature is blocked whatever the 88.6% says. At the assumed $3 per million input and $15 per million output, with 1,500 + 300 tokens a case, the run cost 210 × $0.009 ≈ $1.90, which is why it runs every time the set or the prompt changes.
Wednesday: the boundary call. The PM sent research the 8 failing refund cases on Tuesday, with the question in the form above, and the answer comes back on Wednesday: not a boundary. The model quotes a 14-day window because the 14-day policy is in the system prompt and the 30-day annual-plan exception is not; given the policy text, the class of model in use follows it. This is a context problem, the kind the FDE course's context assembly lesson is about, and the fix is retrieval of the policy section into the prompt.
Thursday: one decision. Retrieve the tenant's policy text per ticket, the FDE ships it in the deployment, and the eval re-runs Friday. The done line does not change: 92% on the set, which stays at 210, and no fact class above 0.5%. The feature fails both until a run says otherwise. That goes in a three-line decision note, with the number.
Friday: 9 of the 10 refund cases pass, the set scores 193 of 210 = 91.9%, and the fact class is down to 2 of 210, the original one and a refund case: 0.95%, still above the 0.5% gate. The feature stays blocked on both the done line and the fact gate. The refund case that still fails is a prorated amount computed from two dates. That one goes back to research on Monday.
When the answer is a boundary
Suppose research had said the opposite on Wednesday: the model class in use does not reliably compute a prorated amount from two dates and a plan price, at any prompt. Then no change the FDE can ship fixes it, and the decision is a product one. The options are the ones every failure mode in this course gets: route that class to the agent without a draft, draft everything except the amount and mark the gap, or accept the error rate and measure undo. For a refund amount in a customer-facing message the first option wins — the "refused" state the design module draws before the screen — and the golden set keeps the cases, scored as pass when the model declines rather than guesses.
The two answers lead to two different owners doing two different things, and the way to find out which is to ask research with cases: "can it handle refunds" gets a shrug; "it fails these 8, here is the policy text, here is the prompt" gets an answer by Wednesday.
What the dashboard cannot tell you
The deployment's monitoring shows the p95, the error rate from the API and the token spend. It cannot show that a draft quoted a 14-day window to an annual-plan customer, because that draft returned 200 OK in 1.8 s and cost $0.009 like every other. Only an agent's edit, an agent's report or a golden-set case catches it, which is why the field report is the first arrow of the week. The latency guardrail still matters — a p95 that drifts past 4 s breaks the done line as surely as a pass rate under 92% — but the p95 and the cost are the numbers a dashboard can own, and the pass rate is the one it cannot.
What each role should never be asked
Do not ask research for a date. A boundary call is a yes, a no or a "with this context, yes"; a delivery date for a capability the model class lacks is not theirs to give.
Do not ask the FDE to generalise. Their note says what they think generalises, and that is an input. Whether it does is the PM's call, made on the sheet with reach and confidence, as the previous lesson works through.
Do not let the PM skip the run. A decision made on the field note alone, before the cases are in the set and the number is in hand, is the week of opinions, and the run costs about $2.
In an interview
The question is "walk me through how you work with engineering and research on a model-backed feature", and the interviewer is testing whether you can name the artefact that connects them. The scored answer names the eval set as the contract, gives the week in the order field report, eval run, boundary call, decision, and puts a number on each step. Module 8's execution round is this week run under time pressure: the pass rate dropped, segment it, decide when to stop diagnosing.
The mistake that loses points: describing the three roles as a meeting. "We sync weekly and research advises us" says nothing about who owns what or what moves between them. The interviewer wants to hear that the FDE's note became ten cases, the cases moved the rate from 92% to 88.6%, research said context not boundary, and the fix shipped by Friday with a number.
Check yourself
1. The FDE's field note reports 60 flags from a week of 240,000 drafts and the engineering lead says 0.025% is noise. Do you write the cases?
Yes, and the lead has the wrong denominator. The report rate measures how often agents bother to click, not how often drafts are wrong; the 14% edit-before-send rate is the error signal, and 14% of 240,000 is 33,600 edited drafts a week. The 60 reports say which class of edit agents mind most, which is what ten golden-set cases should cover. Write them; a $2 run settles it.
2. After Friday's run the set scores 91.9% against a 92% done line, one case short. Ship the retrieval fix to the other tenants or hold?
The deployment it was built for already has it — the FDE owns that. For the product, hold, and say why: 193 of 210 is 0.1 points short of the done line, and the fact class is 2 of 210 = 0.95%, above the 0.5% gate, so the build is blocked on both. The failing case is a prorated amount that may be a boundary, and the honest move is to ask research on Monday; routing that class to a human, scored as a pass when the model declines, would put the set at 194 of 210 = 92.4% and the fact class back to the original one, inside both lines. A done line that bends by 0.1 points this week bends by 2 next week.
3. Research says a failure class is a boundary and the FDE says they fixed it with a prompt change on Thursday. Who is right, and how do you find out by Friday?
Both might be. Research describes the model class in general; the FDE describes one tenant's distribution, where the prompt change may cover the cases that occur. Add 10 cases of the class from a different tenant and run the set. If the prompt holds across tenants, it is context, not a boundary. If it holds only for the first tenant, it is a boundary with a lucky distribution, and the product decision stands.