Discovery and scoping · Scoping

Sizing a pilot from a week of logs

Size the pilot from a week of logs — daily volume, peak rate, requests in flight, cost as a multiple of r — and decide if commercial lines can be excluded.

estimatehardExercise · 3 min · 10 of 64

On this page

The log export arrived. It is five days of timestamps, and the numbers in it are the pilot's size; the one that is not in it is the one that decides the coverage line.

The situation

The claims system owner sent five working days of triage logs: 6,140 claims, each with opened and route-set timestamps and a line-of-business field. Motor is 55% of claims, property 25%, commercial 20%. The median opened-to-route time is 11.2 minutes overall and 19 minutes for commercial. Claims arrive between 08:00 and 18:00, 58% of them between 09:00 and 12:00, and the model's mean response time is about 8 s. At 18:00 on the Friday, 310 claims were open and unrouted; they were cleared on Saturday. The scoping doc's draft definition of done says coverage ≥ 90% of daily volume, p99 ≤ 20 s, and cost per claim ≤ 0.005r. The sponsor has asked whether commercial lines can be left out of the pilot, because the commercial team lead is sceptical.

Assume 0.75 words per token. Each claim is a 1,600-word form and note; the retrieved policy section is 900 words; the instruction prompt is 700 words; the output is a 120-word structured record. The contract prices input at r per million tokens and output at 4r.

The question

Size the pilot — claims a day, the peak arrival rate, the requests in flight against the 20 s p99, and the daily cost as a multiple of r — and decide whether commercial lines can be excluded while the coverage line stays at 90%.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

Volume: 6,140 ÷ 5 = 1,228 claims a day. Peak: 58% of 1,228 is about 712 claims in three hours, 237 an hour, 0.066 a second — twice the all-day average, the peak the workflow map predicted. In flight: Little's Law with the 20 s p99 as residence time gives 0.066 × 20 ≈ 1.3 requests, an upper bound; at a mean nearer 8 s it is about 0.5. One worker and a short queue. The Friday backlog is the burst the average hides: clearing 310 claims in an hour is 0.086 a second, still under two in flight.

Cost: input per claim 1,600 + 900 + 700 = 3,200 words ≈ 4,300 tokens, 0.0043r; output 120 words ≈ 160 tokens at 4r, 0.0006r; per claim ≈ 0.0049r, inside the 0.005r line with little margin. Per day 1,228 × 0.0049r ≈ 6.0r; the customer multiplies by their rate.

Commercial cannot be excluded with the coverage line at 90%, by arithmetic rather than judgement: excluding 20% of claims caps coverage at 80% before a single scanned document or missing policy number is counted. The decision is to keep commercial in coverage — the tool runs on every claim — and measure agreement and time per segment, so the commercial lead's scepticism is answered by commercial's own numbers at the week-6 gate rather than by exclusion. If the sponsor insists, the coverage line is rewritten as 90% of motor and property, in the doc, with the signatures, and the 30% goal says so too. The evidence that would change the decision: a commercial policy set that cannot be extracted to text by day 3, in which case commercial is deferred with a date, not excluded.

Rubric

derives the daily volume and the peak rate from the log, not the average
applies Little's Law with the p99 as an upper bound and says so
converts words to tokens with the stated ratio and prices input and output separately
expresses cost as a multiple of r, never as a currency figure
shows that excluding 20% of claims breaks a 90% coverage line before any other loss
states what evidence would change the exclusion decision

Where answers go wrong

"Average rate: 1,228 ÷ 36,000 s ≈ 0.034 a second." Loses because the pilot serves the morning peak, which is twice that, and the Friday backlog, which is higher still; a pilot sized to the average is slow for the three hours that matter.

"Exclude commercial — it is the hard 20% and the sponsor asked." Loses because the coverage line is then unreachable on the day the doc is signed; an answer that excludes must rewrite the line and say so, and this one did not notice the line.

"Cost: a dollar figure a day, from a remembered per-token price." Loses twice: the figure is unverifiable, and the customer can price 6.0r from their own contract in one multiplication; the time belonged on coverage.