Crawl, walk, run
Size a shadow run from the reviewer's time and the smallest segment, gate each segment on its own reading, and write the rollback trigger before walk starts.
On this page
A deployment reaches the claims floor in three phases. Crawl runs the tool on live claims with nobody looking, to ask whether it is right; walk shows the handler its output, to ask whether it helps; run lets it act unwatched, to ask whether it can be trusted. The gates between them are numbers written before each phase starts, and read per segment.
What each phase measures, and what an error costs
| Phase | The handler | Measured | A wrong output costs |
|---|---|---|---|
| Crawl (shadow) | never sees the output | agreement with the handler's escalate-or-approve call | nothing |
| Walk (assisted) | sees it, decides, can override | time per claim, overrides, misses | seconds; a claim if accepted |
| Run (autonomous) | checks a 1-in-20 sample | errors in the sample | whatever reversing the action costs |
The phases are ordered by the cost of an error, so none is skipped. A tool wrong on 30% of claims costs nothing in crawl; in walk it puts a wrong clause in front of a handler on almost every third claim, and by Wednesday the handlers stop reading the panel.
Crawl: two weeks at 500 claims a day
The shadow run starts in week 4 at 100 claims a day and runs at 500 in weeks 5 and 6, up to the shadow gate a week in the role puts at the end of week 6. Both numbers are derived. Agreement is the definition of done's: the escalate-or-approve call derived from the tool's record, escalate when the route is fraud_review or needs_handler or the claim is over the high-value limit, against the same call derived from the handler's route.
The daily volume is set by the person who reads the disagreements. Agreement against one handler counts that handler's mistakes against the tool, so each disagreement goes to the claims QA lead, who rules on it. She can give two hours a day; at two and a half minutes a ruling, that is 48 rulings. The prompt disagrees on roughly one claim in ten, so 48 rulings cover about 480 claims: call it 500, drawn at random across the day.
The length is set by the smallest segment. A reading near 95% on n claims wobbles by about 2 × √(0.95 × 0.05 ÷ n): ±3.1 points on 200, ±1.4 on 1,000. Commercial, 20% of claims, reaches 1,000 on day 10; the second week exists for it. The same arithmetic is why the gold set cannot gate the rollout: its 40 commercial cases carry ±7 points, and a segment truly at 90% reads 95% or better on 40 cases about one time in five. The gold set gates the prompt; the shadow run gates the segment.
The gate is per segment, not overall
At the end of week 6 the shadow run has 5,000 ruled claims:
| Segment | Claims | Agreed | Agreement | Band |
|---|---|---|---|---|
| Motor | 2,750 | 2,651 | 96.4% | ±0.7 |
| Property | 1,250 | 1,195 | 95.6% | ±1.2 |
| Commercial | 1,000 | 700 | 70.0% | ±2.9 |
| All claims | 5,000 | 4,546 | 90.9% |
An overall gate of 95% fails this pilot at 90.9% and holds back motor and property, 80% of the floor's claims, both above the line. Lower it to 90% and it passes commercial at 70%. A blended number moves with the mix, and whoever wants the rollout most can find a line it clears.
Per segment, the decision writes itself. Motor clears with room; property clears by 0.6 points inside a band of 1.2, so it walks, watched first. Commercial fails by 25 points, for the cause the week-4 reading in the definition of done found on six gold cases: the governing clause sits in an endorsement the index does not hold as a separate document. Commercial stays in crawl on a new index, built in week 5 but held out of the run so the gate read one version, and is re-read at the end of week 8.
Walk, and the trigger that ends it
Walk puts the clauses and a suggested route in the claim screen; the handler decides. It starts on Tuesday of week 7 on the two smaller teams, 12 of the 28 handlers and about 400 motor and property claims a day; the other two teams are the comparison for the week-8 time reading.
The rollback trigger is written before the first suggestion is shown, because a trigger decided after a bad day is a meeting won by whoever wants the rollout most, the failure AARRR and guardrails describes in a launch. The trigger is three misses in a day. An override, about 20 a day at 95% agreement, is walk working. A miss is a claim the handler sent to fraud review on which the tool proposed fast-track.
Three comes from the shadow run. Fraud review is 5% of claims, about 200 of the 4,000 motor and property shadow claims, and the tool proposed fast-track on 5 of them: one in 40. The two teams see about 20 fraud-review claims a day, so they expect half a miss a day, and three or more happens by chance on about 1.4% of days, once in 70 working days. A real break, a wording update never re-indexed, makes several by noon, against half a miss expected in a whole day. An accepted miss surfaces only in the QA lead's weekly blind re-triage of 50 walk claims, which is why the count is low.
Rollback means back to crawl, not off: a per-segment switch hides the suggestions and leaves the tool running unseen, so the evidence keeps accumulating. Claims operations throws it without calling anyone, the way a circuit breaker opens without a meeting.
Run is earned per segment, mostly after you leave
The four shapes reserve run for errors that are cheap to reverse, so nothing in the pilot reaches it inside eight weeks. The one candidate is motor claims under the small-claims limit, corrected the same day when wrong. Its gate is written now: four weeks of walk with handlers accepting the record unchanged on 97% of the segment and the blind re-triage agreeing at that rate, because acceptance measures trust as well as correctness. After the gate, 1 in 20 still goes to a handler. A decline letter never gets a run gate: at 99%, it still sends one irreversible error in a hundred.
The meeting where the customer moves the gate
Monday of week 7. The VP of Claims has read the week-6 update: "91% overall. Make the line 90 and switch the whole floor on today; the COO wants every team using it."
Refusing is wrong: he signed the definition of done, and it is his. Agreeing in the room is wrong too, because it turns a 25-point failure into a launch.
What goes on the table is the arithmetic, in his units. A 90% overall line passes commercial at 70%: about 72 wrong suggestions a day across the floor's 240 commercial claims, shown to the handlers likeliest to accept a plausible one, whose claims take 19 minutes. Against that: motor and property from tomorrow, at the line he signed, which the workflow map showed is enough for the whole 30% target; commercial at the end of week 8 if the new index reaches 95%.
If he still wants commercial now, it is his gate to move. The move goes into the scoping doc with his name and the date, and commercial is reported on its own line every week. The gate belongs to the customer; the arithmetic on the page before they move it belongs to you.
In an interview
"How would you roll this out?" comes up in the case round and the customer scenario. The interviewer listens for three phases with what each measures, a gate with a sample size, read per segment, and a rollback trigger written in advance with the reason for its value. Say that the shadow run's volume comes from the reviewer and its length from the smallest segment.
The mistake that loses points is a single overall gate. It holds back segments that are ready and passes segments that are not, depending on the mix, and the interviewer will ask what the 91% is made of.
Check yourself
1. In week 9 the floor widens walk to all four teams, about 960 motor and property claims a day. Does the trigger stay at three?
No. The floor now sees about 48 fraud-review claims a day and expects 1.2 misses, so three or more happens by chance on about 12% of days: a false rollback every other week. Five in a day brings false alarms under 1% of days, about once in six months. Re-derive it before widening, in the runbook.
2. Commercial's week-8 re-read, on the new index, is 94.4% on 1,000 claims. The deputy says that is close enough. Does commercial walk in week 9?
Not yet. The reading sits inside its band of about ±1.5 points, so the segment may be at 95 or at 93. A week of crawl on all 240 commercial claims a day adds about 1,200 claims and narrows the band to about ±1 point. What would change the answer: misses concentrated in one policy form, in which case the other forms walk and that one waits.