The three-day prototype and the demo
Build a prototype in three days on the first export the customer can produce, fake everything that is not the workflow, and run an eight-minute demo on yesterday's claims.
On this page
The prototype has one job: by Thursday, show the sponsor the workflow working on their own data, with the misses visible. It is not the pilot, it shares no code with the pilot, and the plumbing under it is thrown away the following week. What survives is the prompt, the first fifty labelled cases, and a sponsor who has seen a real claim from yesterday triaged correctly in front of a handler who agreed. The reason to be strict about the three days is that every day spent making the prototype durable is a day the pilot's first real deliverable, the export pipeline, does not get.
Day 1: the export the customer can produce today
Ask for the smallest export the customer can produce in a day without a change request: a CSV of last week's claims from a report they already run, a read-only database view, a folder of 200 PDFs copied by the champion. Do not make the prototype wait for an API, a service account or a data-sharing agreement; those are the pilot's work, raised as tickets on day 1 because they are the long pole, and the subject of getting the data out. What you need is 200 rows with the columns the handler sees, the policy documents for the products those rows cover, and the handler's actual decision on each, which is usually a column in the same report.
The export will be wrong in ways that matter later and not now: three date formats, a customer key that is not unique, a free-text column with the adjuster's note in it. Load it, throw away the columns you do not need, and keep the raw file untouched so the pilot's ETL can start from it. Day 1 ends with 200 claims in a table on an agreed machine (theirs, or a laptop they have cleared for it under a written note of what is on it), and the policy PDFs split into sections by heading.
Day 2: the prompt, the loop, and the first fifty
The prompt is written against the definition of done: a JSON record the downstream system could parse, carrying the route, from which the escalate-or-approve call it scores is derived, and the governing clause, which is scored separately as retrieval. Retrieval is the simplest thing that works on 200 claims: a keyword search over the policy sections, or the whole product's sections in the prompt if they fit the budget from sizing the deployment. Run it over fifty claims in a loop with no retries, no queue, no monitoring, and compare the output to the handler's decision column.
The first number is the prototype's only real output. Fifty claims, say 38 routed to the same desk the handler chose, 41 with the governing clause among the top three retrieved. Those are 76% and 82%, each with an uncertainty of roughly ±12 points at this sample size, and they are what you take to the demo: not as a result, but as the shape of the eval the pilot will build to 200 cases in building the eval set. Spend the afternoon reading the 12 misses. Most prototypes find one systematic cause in the misses (a rider the retrieval never surfaces, a date the form puts in the wrong box) and fixing it moves the number more than any prompt wording will.
Day 3: the screen, and the run on yesterday's claims
The screen is a table: claim id, the handler's actual route, the prototype's route, the top three clauses, and a column for the handler to mark agree or disagree. It is a notebook cell or a static page, not an app. It has no login, because the only people who see it are in the room. Then, on the morning of day 3, run the prototype on the claims logged yesterday, which were not in the export and which nobody has curated.
What to fake and what not to
Fake the plumbing: the export instead of ETL, a keyword search instead of an index, a loop instead of a queue, a laptop instead of a deployment, the hosted API whatever the eventual placement from where the model sits. Fake the UI. Fake auth entirely, by keeping the data on a machine the customer has cleared.
Do not fake the data, the output format, or the decision column. A prototype on synthetic claims shows that the model can read a claim, which the sponsor already believes; a prototype on their claims shows that it can read theirs, with their riders and their adjusters' shorthand, which is the only thing in doubt. Do not fake the misses: a prototype that shows only the hits is a demo of your prompt-writing, not of their workflow.
Why the demo runs on their data and never on the sample
The sample is the 30 claims the champion sent in week 0, and it is clean: the champion picked ones they understood. A demo on the sample scores 29 out of 30, the sponsor repeats "97%" to the board, and week 3 on live claims delivers 71%, at which point the pilot is explaining a drop rather than reporting progress. The demo runs on yesterday's claims, selected by the handler in the room, because the number it produces is the number the pilot will produce, and the sponsor who sees 3 out of 4 with the miss explained has a true expectation to carry out of the room.
The eight-minute demo script
Minute 0 to 1: the number on the wall. "Eleven minutes a claim, seven of them finding the wording. That is the number this attacks." No slides about the model.
Minutes 1 to 4: three of yesterday's claims. The handler picks them. Open each, show the three clauses and the proposed route, and let the handler say whether it is the clause they chose. Say nothing while they decide.
Minutes 4 to 6: the one it got wrong. Open a miss you found that morning, show the wrong clause, and say why: "the exclusion is in a rider the retrieval never saw; the pilot's index fixes that, and the eval will count every case like it." The sponsor is watching to see whether you hide failures. You do not.
Minutes 6 to 7: the planted question. Before the meeting, ask the sponsor to ask one question at this point: "what happens when it is wrong?" Planting it is not theatre; it is making sure the one question every stakeholder has gets asked by the person with the most authority, and answered in the room. The answer is the shape from the four shapes: a handler signs each record, nothing moves without a signature, and the 200-case eval sets the gate before any segment runs unassisted.
Minutes 7 to 8: what the pilot needs next. The export pipeline, about two weeks (the view's two tickets take nine working days), and the first row landing is when the pilot's clock starts. The sponsor leaves with a date, not a feeling.
In an interview
The deep-dive and the customer-scenario rounds both reach for this: "you have three days before the sponsor meeting; what do you build?" The strong answer names what it fakes and what it refuses to fake, puts the data request on day 1, and ends with a demo on uncurated data that shows a miss on purpose. Say the fifty-case number with its uncertainty, and say what the pilot will do to turn it into a 200-case result.
The mistake that loses points is demoing on the sample. It reads as either naivety (you did not know the sample was curated) or salesmanship (you did), and the interviewer has watched pilots die of the 97% that became 71%. The second mistake is spending the three days on infrastructure: a queue, a deployment and a login by Thursday, and no number, because the export was requested on day 2.
Check yourself
1. On day 2 the fifty-claim run gives 38 of 50 on route. The champion suggests re-running on the 30-claim sample "to get a better number for the VP". Decide.
No. 38 of 50 is 76% with roughly ±12 points of uncertainty; 29 of 30 on a curated sample would be a different population, not a better estimate, and it would set the expectation the pilot has to miss. Take 76% to the demo as the shape of the eval, show the systematic cause in the 12 misses, and ask the champion for yesterday's claims instead. The evidence that would change this: the sample turns out to be a stratified draw from last month's claims, in which case it is an eval set, and the number from it is worth reporting with its size.
2. The customer cannot produce any export without a four-week change request. What do you build in the three days?
Build on what a person can produce by hand: the champion copies 50 claims and their policy PDFs into a shared folder on day 1, and the handler's decisions come from the same screen, typed into a column. Fifty is enough for the demo and the first eval shape. Then say in the demo's last minute that the four-week change request is the pilot's first deliverable and the clock starts when it lands, which moves the sponsor's attention to the thing only the sponsor can speed up.
3. The demo went well and the sponsor asks to roll the prototype out to all 28 handlers on Monday. Estimate what breaks first.
The prototype has no queue, no fallback and runs on a laptop with a one-off export, so Monday's claims are not in it at all: 28 handlers would be looking at last week's 200 rows. The first thing that breaks is data freshness, before any model problem. Say that, and offer the shadow run from crawl, walk, run as the first rollout step: once the export pipeline lands, the shadow run: the pilot beside the handlers on live claims with nobody acting on it yet, 100 a day in week 4 and 500 a day in weeks 5 and 6, gated per segment at the end of week 6.