Map the workflow before the model
Draw a claims floor's as-is workflow with a count on every step, then pick the one step a model replaces and the one it must not take over in the pilot.
On this page
The model goes into one step of a workflow that already exists, and until that workflow is drawn with a count on every step, "AI for claims triage" is a sentence rather than a plan. The map is the first thing you produce that the sponsor can argue with, and the arguing is where the pilot's shape gets decided. It takes a day of discovery and a week of logs, and it answers two questions the scoping doc cannot be written without: which step the model replaces, and which it must not.
The as-is, with counts
The claims floor from the discovery interviews, drawn as the handlers run it:
The counts on the diagram come from three sources and are labelled by source. Volume — 1,200 first-notice-of-loss claims a working day, 60% of them between 09:00 and 12:00 — is the team lead's figure, confirmed from the claims system. Headcount, 28, is the team lead's. The step timings are the median of a week of per-claim timestamps: claim opened, first document search, route set. The routes and their shares — fast-track about 55%, assign to an adjuster about 40%, fraud review about 5% — are from the same week of logs.
Capacity falls out of the counts. 28 handlers at roughly 450 productive minutes a day is about 12,600 handler-minutes; 1,200 claims at a median 11 minutes is 13,200. The floor runs about 5% short, so roughly 55 claims a day carry over, and the backlog is cleared on Saturday overtime. That arithmetic is the sponsor's "30% faster" translated into the floor's terms: cut the median to 7.7 minutes and the day needs 9,240 handler-minutes, the overtime goes, and about a quarter of the floor's time is free for the claims that need it. The sponsor has probably never seen the 13,200; showing it is how you earn the right to change the build.
Where the minutes are
A median hides a shape, and the shape decides what the pilot can promise. The 7-minute search step is a median across all lines; by segment it is about 5 minutes for motor, 7 for property and 15 for commercial — 19 for a commercial claim end to end — where the policy set is three times longer and the governing wording is often in an endorsement rather than the base policy. Commercial is 20% of claims, against 55% motor and 25% property, and roughly 40% of the search minutes.
Which means two things. The pilot's 30% can be reached on motor and property alone, and a pilot that reports one number across all lines will be pulled down by commercial even when it is working. And the handler's experience of the tool is set by the slow cases, not the median, for the same reason p99 is what users feel in any system: a tool that finds the clause in 2 seconds for motor and fails on commercial is, to the handler who does commercial, a tool that fails. The map records the segment split so the scoping doc can measure per segment, and each segment's gate can be read on its own.
The step the model replaces
The naive placement comes first, because it is what the kickoff asks for. The sponsor's picture is a model that reads the claim and sets the route: the 4-minute decision step, automated. It is the most impressive demo and the worst first pilot, for three reasons that are visible on the map.
The decision is expensive to check. A route is right or wrong only by a senior handler re-doing the triage, 11 minutes a claim, or by an adjuster's or reviewer's verdict weeks later, so the pilot can afford about 200 checked routes: enough to score a drafted route, not enough to let one through with nobody signing. Every error is expensive: a fraud-review claim fast-tracked is a paid claim that should not have been, and that route is 5% of volume — the model can be 95% right by never choosing it. And the step is regulated: the handler's name is on the route, and the customer's compliance team has a view on a model's name being there instead.
The 7-minute step has none of these problems. Finding the governing clause is retrieval: the handler types a policy number, opens the PDF, searches, scrolls, reads. The model's job is to surface the two or three candidate clauses inside the screen the handler already has open. The handler checks every suggestion by reading it, and a first label exists for free — the clause the handler cited is in the claim record — so a first eval can run on past claims before the integration exists; the gold set proper is two senior handlers' labels, adjudicated. An error costs seconds: the handler sees a wrong clause and searches as before. And the handler's name stays on the route.
So the pilot replaces the search and leaves the decision. The map now has one box with an accent on it, and the scoping doc's goal line follows from that box: the median search step from 7 minutes to 3.5, half, which takes the claim to 7.5 and clears the sponsor's 7.7.
The step it must not replace
"Not in the pilot" needs to be written down as firmly as "in", because the decision step will come up again in week 5 when retrieval is working and the sponsor asks why the model cannot just set the route. The answer is in the definition of done: the pilot's agreement number is read on 200 adjudicated cases, which can show that a drafted route is usually right and cannot show that the rare expensive error is rare enough to go unwatched. So the tool drafts the route and the handler signs every one, for the whole pilot. Setting a route with nobody signing waits for the run gate after the pilot, for small motor claims only, after four weeks of assisted use with handlers accepting the draft unchanged on 97% of them, as crawl, walk, run sets out. That is a sentence for the out-of-scope section, with the condition that would bring it back.
The other step the pilot leaves alone is intake. It is tempting to tidy the form while you are there; the form belongs to a different system owner, and a pilot that needs two systems to change needs two sponsors.
Getting the counts
The map is only as good as its timestamps, and the first week of logs is usually not quite the week you asked for. Three things to check before any number goes on the diagram. The claims system's "opened" event may fire when a claim is assigned, not when the handler starts work, which inflates the search step by the queue wait; compare the log median against the six claims you timed by hand, and if the log says 14 minutes where you watched 11, the log is measuring something else. The document system may not log searches at all, in which case the search step is the gap between "opened" and "route set" minus the observed decision time, and the diagram says so. And the week has to be a normal one — a week with a storm in it has the volume and the segment mix of a storm.
The exercise sizing a pilot from a week of logs works through one such week, including the Friday backlog the daily average hides. Volume arithmetic — claims per day to claims per hour at the peak — is the back-of-envelope method from the system design course, applied to a floor rather than a service.
In an interview
The case round gives you a workflow in prose and watches whether you draw it with counts before you place the model, and whether the place you choose is justified by the counts rather than by what would demo best. Say the capacity arithmetic out loud — 28 handlers, 450 minutes, 13,200 needed — because it turns the sponsor's percentage into a number of minutes and shows you know what the goal costs. Then name the step, say why it is cheap to check, and say which decision stays with the handler and what would bring it back.
The mistake that loses points is placing the model at the decision step because it is the most impressive, without noticing that it is expensive to check, that its errors are the expensive kind, and that the retrieval step next to it has all three properties you need. The interviewer will ask how you would know in week 4 whether the routes are right, and the answer "we would check some" is the end of the round.
Check yourself
1. The week of logs shows the search step at a median of 4 minutes, not 7. Does retrieval still carry the pilot?
Not alone. Taking 4 minutes to 2 brings the claim from 11 to 9, an 18% cut against a 30% goal. Either the goal moves — the sponsor accepts 18% on the search step with a second phase on the decision — or the pilot has to touch the decision step after all, with its labelling problem. Go back to the sponsor with both options and the arithmetic; do not quietly keep the 30%.
2. Commercial lines are 20% of claims and 40% of search minutes. The sponsor wants one number for the whole floor. What do you measure, and what do you report?
Measure per segment, report both. The pilot's agreement and time numbers per segment are what tell you whether the tool works; the floor's median is what the sponsor's 30% is written against, and it hides the shape. If motor and property hit the target and commercial does not, the floor's median can still come close to 7.7, because commercial's claims sit at the slow end either way, but the floor's handler-minutes fall only about 20%: commercial carries about a third of them (0.2 × 19 = 3.8 of about 11 minutes a claim), so the cut is about 0.65 × 30% ≈ 20%, some 10,600 handler-minutes a day rather than 9,240. Say both in week 1, so neither number in week 8 is a surprise.
3. At the week-6 gate shadow agreement with the handler is 96% on motor claims, and the sponsor asks to let the tool set fast-track on motor with nobody signing. Do you agree?
Not in the pilot. Motor is about 660 claims a day and fraud review about 5% of them, 33 a day; a draft that proposes fast-track on one fraud-review claim in 40, a rate a 96% figure can easily hide, sends nearly one claim a day towards payment with nobody looking. Shadow agreement shows the draft is usually right, not that its rare expensive error is rare enough to go unwatched. So the draft goes to handlers to sign in weeks 7 and 8, and an unsigned route waits for the run gate after the pilot: small motor claims only, after four weeks of assisted use with the draft accepted unchanged on 97% of them and a blind re-triage agreeing. Point at the out-of-scope line that already says so.