Architecture, sizing and the first prototype · Shapes and sizing

Token budget for the triage queue

Size the claims-triage pilot in tokens per claim and per day, find the peak rate and the requests in flight, and decide whether the four worked examples stay in the prompt.

estimatemediumExercise · 3 min · 25 of 64

On this page

The scoping doc needs a sizing paragraph by Friday, and the sponsor has asked whether the four worked examples in the prompt are "worth what they cost".

The situation

The regional insurer from discovery: 1,200 first-notice-of-loss claims a working day, triaged by 28 handlers. Each claim is a 3-page form plus a 1-page adjuster note, about 1,600 words. The retrieved policy section is about 900 words; the instruction prompt is 700 words, 500 of them four worked examples; the output is a JSON triage record of about 120 words. The contract prices input at r per million tokens and output at 4r. Claims arrive 08:00–18:00, 60% of them between 09:00 and 12:00. The definition of done sets p99 latency at 20 s and cost per claim at 0.005r or under. Assume 0.75 words per token, a mean model residence time of 8 s, and a two-reviewer agreement noise band of about 2 points on the 200-case eval set.

The question

Give the token budget per claim and per day, the peak rate and the requests in flight, and decide whether the four worked examples stay in the prompt.

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

At 0.75 words per token, input per claim is 1,600 + 900 + 700 = 3,200 words ≈ 4,300 tokens; output ≈ 160 tokens. Per day: 1,200 × 4,300 ≈ 5.2M input tokens and 1,200 × 160 ≈ 0.19M output. Cost per day ≈ 5.2r + 0.19 × 4r ≈ 6.0r, so ≈ 0.005r per claim, on the definition-of-done line. The answer stops there: r is in the contract and the customer multiplies.

What the estimate shows is structure. Cost is linear in volume and 87% of it is input, and 77% of the input is the customer's own form and policy text. The only lever the prompt author holds is the 940 tokens of instructions and examples, about a fifth of the bill.

Peak: 720 claims in 3 hours = 240 an hour ≈ 0.067 a second, about 17k input tokens a minute against the rate limit. Little's Law takes the mean residence time: 0.067 × 8 s ≈ 0.5 requests in flight. Using the 20 s p99 instead gives 0.067 × 20 ≈ 1.3 as an upper bound. One worker with a small queue either way, which is the arithmetic sizing the deployment walks through.

The four examples are about 670 tokens, 15% of input, so cutting them to two saves about 0.4r, 7% of 6r a day. That is not a reason to cut them. Keep them unless the 200-case eval shows they lift agreement by less than the two-reviewer noise band of about 2 points, in which case cut to two and re-run. The constraints that decide this pilot are agreement and latency; the cost number's job was to show that cost will not.

Rubric

converts words to tokens with a stated ratio, not a remembered one
separates input from output and prices them differently
expresses cost as a multiple of r, never as an invented currency figure
computes the peak rate, not the daily average, and uses the mean residence time in Little's Law with p99 as a bound only
ties the few-shot decision to the eval and the noise band, not to cost
names agreement and latency as the constraints that actually bind

Where answers go wrong

Quoting a per-token price from memory and producing an absolute figure. Loses twice: the figure is unverifiable, and the customer can price 6r from their own contract in one multiplication; the answer's time belonged on agreement and latency.

"Drop the examples, they are 15% of the input." Loses because about 0.0007r a claim is nothing the sponsor will notice, while a 3-point drop in agreement is a failed definition of done; the decision belongs to the eval.

Sizing concurrency from the daily average and the p99 together: 1,200 ÷ 36,000 s × 20 s ≈ 0.7. Loses because it mixes a rate that understates the peak with a residence time that overstates the mean; the number happens to be small here, and the method gives the wrong fleet the moment a backfill arrives.