Discovery and scoping · Scoping

Sign this scoping doc?

Read a draft scoping doc for the triage pilot, decide whether to sign it tonight, and name the three lines that change before the VP sees it.

critiquemediumExercise · 3 min · 11 of 64

On this page

A draft scoping doc is on your desk with your name on the signature line, and the VP's meeting is tomorrow. Signing it is a decision about week 8.

The situation

Your commercial lead has drafted the scoping doc for the claims-triage pilot from the kickoff notes and asks you to sign it tonight so the VP of Claims can sign tomorrow. It is nine pages: two of text, then an architecture section, an integration appendix and a risk register. The two pages of text, condensed:

Goal:        Improve claims triage with AI.
Scope:       An AI assistant for handlers. Details TBD with the customer.
Data:        Access to the claims system and the policy documents.
Done when:   Handlers are satisfied with the assistant; accuracy >= 99%.
Timeline:    8 weeks from kickoff.
Signatures:  [commercial lead]   [FDE]

From discovery you know: 1,200 claims a day, 28 handlers, a median 11 minutes of which 7 are the policy search; the contract promises 30%; the claims QA lead will own a 200-claim gold set labelled by two senior handlers, and in her last calibration round two handlers agreed on 87 of 100 claims and, once the Head of Claims Operations had ruled on the 13, a blind re-label matched the rulings on 97; and the claims system owner has not yet committed to a date for the log export. The VP has told your commercial lead that the board wants to see progress by the end of the quarter.

The question

Do you sign it tonight, and which three lines do you change before the VP sees it?

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Not tonight: signed as drafted, it commits week 8 to a 99% line nobody can reach and a survey that cannot fail. The three lines that change first:

The goal gets its number and its mechanism: "Cut median first-triage time from 11 minutes to 7.7 or less (30%) on 1,200 claims a day, by surfacing the governing policy clauses inside the handler's claim screen." "Improve with AI" cannot fail in week 8, which means it cannot succeed either.

The done-when line becomes the five lines of the definition of done, and the 99% goes. Even adjudicated, a careful re-label agrees with the gold on about 97 in 100; a tool cannot be scored above the people who built its gold set, and a line at 99% is unreachable by construction. Agreement ≥ 95% on the 200-case set, p99 ≤ 20 s, cost ≤ 0.005r, coverage ≥ 90%, a named owner after handover.

The signature block gains the VP of Claims, the Head of Claims Operations, who owns the floor's time, the claims QA lead and the claims system owner. Without the QA lead's name the gold set has no owner; without the system owner's, the data has no date — and the timeline changes with it, to "8 weeks from the first row of log data", the only honest start while the export has no date.

The seven pages of architecture, integration and risks move to your design doc, referred to in one line; what remains is the two pages the scoping doc describes. Evidence that would reverse the agreement number: if this pilot's blind re-label agrees on 99 of 100, the ceiling is near 99% and the customer's figure stands.

Rubric

declines to sign and says what signing would commit the pilot to
replaces the goal with a number, a baseline and a mechanism
sets the agreement line from the reviewers' own agreement ceiling, not the customer's wish
adds the signatures that give the gold set and the data an owner
moves the engineering sections out of the document the customer signs

Where answers go wrong

"Sign it; the details get fixed in week 1." Loses because the VP signs the same document, and in week 5 the only agreed definition of done is a satisfaction survey and 99%; the engineer is then negotiating against their own signature.

"Keep 99% — the customer asked for it, and aiming high is fine." Loses because the ceiling is a careful reviewer's agreement with the adjudicated gold, about 97%; a pilot measured against 99% fails in week 8 while working, and the renewal case is lost on a number the labellers themselves could not reach.

"Rewrite it properly: a twelve-page statement of work with the architecture in the body." Loses because the sponsor reads two pages and signs the rest unread; the out-of-scope list the pilot depends on is then on a page the VP never agreed to.