Discovery and scoping · Discovery

The kickoff that asked for a chatbot

Decide what to build first when the sponsor asks for a chatbot and the contract promises 30% faster triage, and say it to the VP in the room.

casemediumExercise · 3 min · 8 of 64

On this page

The sponsor has asked for a specific thing, the contract promises a number, and the two are not the same. What you agree to in the first hour sets what week 8 can show.

The situation

A regional insurer receives 1,200 first-notice-of-loss claims a working day. Twenty-eight handlers do first triage at a median 11 minutes per claim: 7 minutes finding the governing wording in a 340-page set of policy PDFs, 4 minutes deciding the route — fast-track, adjuster or fraud review. The deal, signed last month, promises "30% faster triage" in an 8-week pilot with one forward-deployed engineer, you. The handlers work in a claims system with a fixed screen layout that the IT owner, who is in the room, can add a panel to. Sales is on the call. The VP of Claims opens the kickoff with: "We want a chatbot the handlers can ask. They should be able to type a question about the policy and get the answer." Two handlers you spoke to that morning said the PDF search is "most of the job"; one of them keeps a personal index of common clauses in a spreadsheet. Nobody has mentioned an eval, a gold set, or how the 30% will be measured.

The question

What do you agree to build first, and what do you say to the VP in the room?

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Separate the ask from the outcome. The outcome in the contract is 30% off 11 minutes: a median of 7.7 minutes or less. The 7-minute step is retrieval and the 4-minute step is judgement, so the minutes are in the search, as the workflow map shows. Surfacing the two or three governing clauses inside the claim screen the handlers already use attacks the 7 minutes directly: halving it on the 95% of claims where the panel finds the clause saves 3.3 minutes, exactly the 30%, with no new interface, no free-text behaviour to evaluate, and a first label that already exists, since the clause the handler used is in the claim record. A chatbot is one way to the number; this is a faster one, and the chat surface can follow the week-6 gate if handlers ask for it after using the panel.

So the first build is the in-screen clause panel, a 200-claim gold set, a shadow run on live claims from week 4, and a definition of done the VP signs: agreement ≥ 95% with the adjudicated gold and a retrieval hit rate ≥ 95%, both read by the claims QA lead at the week-6 gate, and median triage ≤ 7.7 minutes in the assisted weeks 7 and 8. To the VP: "Your number is the goal. Here is the shortest path to it, and here is the date you will know whether it worked." What would make the chatbot the right first build: if the log shows the search step is already under 3 minutes and the handlers' time is going to questions the policy set does not answer — then the minutes are elsewhere and the retrieval panel cannot reach the number.

Rubric

separates the requested solution from the promised outcome
uses the 7/4 split to locate where the minutes are
proposes a measurable definition of done with a date and a measurer
keeps the sponsor's goal intact while changing the build
says what would have to be true for the chatbot to be the right first build

Where answers go wrong

"The customer knows their business — build the chatbot." Loses because it has no eval: at week 8 nobody can attribute the 30% to anything, and the renewal case is an anecdote.

"The 30% is a sales number, so the first job is to reset expectations." Loses because the sponsor's number is the contract, and an engineer who opens by renegotiating it has spent the goodwill the pilot runs on before doing any arithmetic; the 7/4 split shows the number is reachable, so there is nothing to reset.

"Build the chatbot, and add a retrieval panel later if it is slow." Loses because the chatbot has to be evaluated on free-text answers nobody has a rubric for yet, and the measure in the contract is triage time, not answer quality; the panel is the thing the measure can see.