Definition of done for an AI deployment
Write the five-line definition of done — agreement, p99, cost as a multiple of r, coverage, a named owner — with the arithmetic that sets each number.
On this page
A definition of done for an AI deployment is five lines, each with a number, a way of measuring it, and the name of the person who reads the measurement. The customer signs it in week 0 and it is read out in week 8; nothing else in the pilot, apart from the goal line it qualifies, is "done" in any sense that the renewal conversation can use. The five lines for the claims-triage pilot, and the arithmetic behind each number, are below. The shape transfers to any deployment; the numbers do not, and the lesson is mostly about where the numbers come from.
The five lines
- Agreement ≥ 95% with the adjudicated gold labels on the customer's 200-case set: for each claim, escalate to a senior handler or approve, labelled by two senior handlers and owned by the claims QA lead.
- p99 latency ≤ 20 s from claim open to clauses on screen.
- Cost per claim ≤ 0.005r, where
ris the contracted per-million-token input rate. - Coverage ≥ 90% of daily claim volume.
- A named owner at the customer for the prompt, the eval and the on-call, from handover at the end of week 8.
The goal line above them in the scoping doc carries the outcome — median triage time against the 11-minute baseline, measured in the assisted weeks; the five lines are the conditions under which that number counts. A decision whose errors are not symmetric can earn lines of its own by amendment, signed by whoever owns that risk; the head of underwriting's two escalation lines in score the eval set are the example.
Each line is derived, not chosen, and a reader who knows where the number came from can defend it in the room when the sponsor asks why not 99.
Agreement: the ceiling is the customer's own reviewers
95% is set from the top down. Two senior handlers labelling the same 200 claims escalate or approve will not agree with each other every time: on this floor they agree on 174 of 200, 87%. Once the champion, the Head of Claims Operations, rules on the 26 disagreements, a blind re-label of 100 agrees with the adjudicated gold on 97, and that 97% is the ceiling. A tool cannot be scored above the agreement of the people who built its gold set — a reading above that ceiling is measuring the labellers' noise, not the model — so the line sits just under the ceiling, and the gap is the noise band. The line is written in week 0 against the ceiling you expect and checked when the labelling ends; the module on evals works through the labelling, the adjudication of the disagreements, and what happens to the line if the ceiling comes back lower. Here the point is that the number is the customer's, taken from their reviewers, not a figure from a leaderboard.
The 200 matters too. On 200 cases one case is half a point, so a reading of 94% against a line of 95% is a two-case difference, and the doc says how a two-case difference is adjudicated — the QA lead looks at the cases — rather than leaving it to be argued. Fewer than about 100 cases and the noise band swallows the gate; more than about 300 and two senior handlers giving two and a half hours a day cannot double-label them by week 4: at 11 minutes a label, 300 × 2 × 11 minutes is 110 reviewer-hours, 22 working days each.
Latency: p99, because the handler feels the slow one
20 s is the handler's tolerance, observed rather than assumed: it is roughly the point at which, during the prototype sessions, a handler stops waiting and opens the PDF by hand, which turns a claim with a tool into a claim without one plus 20 s. The percentile is p99 for the reason tail latency gives for every user-facing system — a median of 8 s with a p99 of 40 s is a tool that fails on one claim in a hundred, which on a 1,200-claim day is twelve claims on which a handler gives up and tells their neighbour.
Little's Law says what the gate costs in capacity. At the morning peak, 60% of 1,200 claims arrive in three hours: 720 ÷ 10,800 s ≈ 0.067 claims a second. Using the 20 s gate as the residence time gives 0.067 × 20 ≈ 1.3 requests in flight, an upper bound, since the mean is nearer 8 s and 0.067 × 8 is about 0.5. One worker and a short queue. The Little's Law lesson has the derivation; the sizing module applies it to the whole pipeline.
Cost: a multiple of the rate, never a figure
The cost line is written as a multiple of r because the rate is in the contract and a currency figure in the doc is wrong the day the contract is renegotiated. The arithmetic, at the course's assumption of 0.75 words per token:
- Input per claim: a 1,600-word claim form and note, a 900-word retrieved policy section, a 700-word instruction prompt — 3,200 words ≈ 4,300 tokens, priced at
rper million: 0.0043r. - Output per claim: a 120-word structured record ≈ 160 tokens, priced at
4r: 0.0006r. - Per claim ≈ 0.0049r; the line is set at 0.005r with the rounding as margin.
- Per day at 1,200 claims: about 6r. Per year at 250 working days: about 1,500r.
The customer multiplies by their r and has the number in one step; the doc never ages. The line also tells you what the lever is: input is seven-eighths of the cost, so the only way to move it is the prompt, and the only reason to grow the prompt is agreement. A 600-word addition of worked examples is roughly 800 tokens, 0.0008r more per claim, which breaks the line at 0.0057r — so examples go in only if they replace words already in the prompt and the eval shows they lift agreement by more than the noise band; adding them on top means renegotiating the cost line in writing first.
Coverage: the denominator guard
Coverage is the line most often left out, and the one that makes the other four honest. It is the share of the day's claims for which the tool returned clauses at all: claims received in the denominator, claims served in the numerator. Without it, agreement is computed on the claims the tool chose to answer, and a tool that answers the easy 40% of claims at 99% agreement reads as a success while the handlers do the other 60% by hand.
The 10% that is allowed to fall through has names: scanned policy documents the text extraction cannot read, claims with no policy number at intake, and the tool declining to answer where the retrieved sections conflict. Each is measured, and each has a different owner — the scanned documents are the document system owner's, the missing policy numbers are intake's, the declines are yours. Agreement is reported on covered claims and coverage is reported beside it, always both, so the sponsor never sees one number that could have been improved by narrowing the other.
A named owner from handover
The fifth line has no number, and it is the line that decides whether the pilot was a pilot or a demo. A name — the QA lead, usually, or a handler who became the tool's informal expert during the assisted phase — who from handover at the end of week 8 owns re-running the eval when the prompt changes, approving a prompt change, and being the first call when the queue stops. A team is not a name; "claims operations" owns nothing at 02:00.
The line sits in the definition of done rather than in the handover plan because the customer has to agree to it in week 0, when the person can be identified and their time negotiated, not in week 7 when the sponsor discovers that ownership was assumed. The scoping doc carries it with the other four, and it is the line the sponsor most often tries to leave blank.
Reading the five lines at week 4
The diagram's week-4 column shows a pilot that is working and not done. Agreement at 91% is 127 of the 140 gold cases labelled so far, six cases under the line of 133; of the 13 disagreements, the QA lead's look shows six are commercial policies where the governing clause is in an endorsement — a retrieval problem with a known fix, index the endorsements separately, worth exactly the six cases the line needs and no margin. Coverage at 84% is six points under, and the shortfall is almost entirely scanned documents, which is the document owner's problem and not a model problem at all. Latency and cost are inside their lines with margin, and nobody has named an owner. Three different conversations, with three different people, and the five lines are what tell you which is which.
In an interview
The deep-dive and the case round both ask for a definition of done, and the interviewer is checking that every line has a number, that you can say where the number came from, and that one of the lines is a denominator. Give the five lines in order, then derive two of them out loud — the agreement ceiling from the reviewers, the cost from the word counts and the rate — and say that the cost is a multiple of r because the contract owns the figure.
The mistake that loses points is a definition of done without a coverage line, so that every other number can be met by answering fewer claims. The interviewer will ask what the agreement figure is a percentage of; "the claims the tool handled" is the wrong answer, and once it is said, the agreement line is worth nothing.
Check yourself
1. Week 5 reading: agreement 93.5% on the 200-case set. The sponsor asks whether the pilot is on track. What do you say?
93.5% is 187 of 200, three cases under the line of 190. Say so in cases, not points, and say what the 13 disagreements are: if most are the same commercial endorsements as week 4, the fix is already in flight and the per-segment reading at the week-6 gate is the one that counts; if they are new failure types, the line is in doubt and the sponsor should hear that now. A three-case gap is inside the noise band, so the adjudication clause applies — the QA lead looks at the 13 — and the answer is "on track, with those 13 cases on the table on Thursday".
2. The team proposes adding 600 words of worked examples to the prompt. Agreement on the eval moves from 91% to 93%. Does the change go in?
No, as proposed. 600 words ≈ 800 tokens, which adds about 0.0008r per claim and takes cost per claim from 0.0049r to 0.0057r — over the gate. And a two-point lift is four cases on 200, inside the noise band, so it may not be a lift at all. Either make room first — cut as many words from the existing prompt as the examples add, which keeps the claim near 0.0049r — or show the lift on a second 200-case sample and take a change to the cost line to the signatories. The decision belongs to the eval and the cost line together, not to either alone.
3. The p99 reads 24 s and the median reads 6 s. What do you change first: the model, the retrieval, or the gate?
Neither the model nor the gate. A median of 6 s with a p99 four times higher is a tail problem, and the tail is usually one slow component — the commercial policy set is three times the size, and retrieval over it takes longer. Measure p99 per segment; if commercial carries the tail, the fix is the index, not the model, and the 20 s gate was right. Move the gate only if the per-segment numbers show the tail is in the model's own response time on ordinary claims.