Architecture, sizing and the first prototype · Prototype and demo
The demo that ran on the sample data
Find what the week-2 demo got wrong, work out what its 29-of-30 was worth, and decide what to say and change after the sponsor has told the board 97%.
critiquehardExercise · 3 min · 26 of 64
On this page
The demo was a success, the sponsor has repeated its number to the board, and a week later the pilot is explaining a drop instead of reporting progress.
Week 2 of the claims-triage pilot. The FDE built the prototype in three days on the 30-claim sample the champion had sent before kickoff, chosen by the champion as "good representative cases". The demo opened all 30 on screen, one after another: 29 routed to the desk the handler had chosen, with the governing clause in the top three. No misses were shown, because there was only one and it looked like a labelling error. The VP of Claims told the board the next day that the pilot was "at 97%". Week 3: the export pipeline lands, the prototype runs on 1,100 live claims from the previous day, and agreement on route is 71%. Half the misses are commercial claims, which were 2 of the 30 in the sample and are 20% of daily volume. The VP has asked for a meeting on Monday "to understand the drop". The definition of done says agreement ≥ 95% on the customer's 200-case set.
Critique the demo as it was run, estimate what the 29-of-30 was actually evidence for, and decide what you say on Monday and what changes in the pilot plan.
The demo's defect is the population, not the prompt. Thirty cases chosen by the champion for being understandable is a sample of the champion's confidence, and 29 of 30 has a 95% interval of roughly 83–99%; the honest reading was "above 83% on easy cases". It under-represented commercial claims threefold (2 of 30 against 20% of volume), and the one miss was explained away rather than shown. So the 97% was evidence that the prompt works on simple motor claims. The three-day prototype demos on yesterday's uncurated claims for this reason.
On Monday, say that first, with the arithmetic: "the 30 were chosen, not drawn; 29 of 30 on a chosen set is consistent with anything above 83%, and the 71% on 1,100 live claims is the first real number." Then split the 71% by segment: about 160 of the 319 misses fall on 220 commercial claims, so commercial is near 27% and the rest near 82%, still short of 95%. Then reset the expectation: the definition of done's 200-case set, stratified by segment, is the number that matters.
The plan changes in three places. The eval set is drawn this week, stratified so commercial has its 20% share, then double-labelled and adjudicated over about three weeks. The rollout gate becomes per segment: motor and property may reach 95% by the week-6 gate; commercial gets its own retrieval fix and its own gate. And the weekly update carries the 71% with the segment split, so the board's next number comes from the eval, not a demo.
What would reverse this: if that day fell in a storm week, the live sample is skewed too, and the per-segment eval is still the answer.
"Improve the prompt until live agreement is back at 97%." Loses because 97% was never real; chasing it on 1,100 live claims without a stratified eval set means the next number is as untrustworthy as the first, and the board hears a second drop.
"Tell the VP the demo was always a demo and the 71% is normal for week 3." Loses because the VP said 97% to the board on the strength of what you showed; the engineer owns the expectation they set, and an answer without the segment split and the eval plan leaves the VP with nothing to take back upstairs.
"Exclude commercial claims from the pilot scope so the number recovers." Loses because they are 20% of daily volume and the definition of done promises 90% coverage; cutting them moves the pilot below its own done line to make a dashboard look better, and the blocker will notice before the sponsor does.
