Sizing the deployment
Turn one document into tokens, one day into a token budget and a peak rate, then derive concurrency with Little's Law and cost as a multiple of the contracted rate.
On this page
Sizing an AI deployment is five multiplications, done in a fixed order: words to tokens for one document, tokens per document to tokens per day, the day to its peak hour, the peak to requests in flight, and tokens to cost as a multiple of the contracted rate. Each step uses a stated assumption so the customer can swap in their own, and the whole thing fits on half a page of the scoping doc. The order matters because every later number is wrong if the first one is a guess.
One document, in tokens
Assume 0.75 words per token for English prose, which is the same assumption as 1.3 tokens per word and roughly four characters a token. Say the number out loud as an assumption every time; the customer's documents may be tables, codes and amounts, which tokenize worse, and a day-one measurement on fifty real documents replaces the assumption.
The triage call for the insurer from map the workflow before the model has four input parts and one output:
| Part | Words | Tokens at 0.75 words/token |
|---|---|---|
| Claim form (3 pages) plus adjuster note | 1,600 | ≈ 2,130 |
| Retrieved policy section | 900 | ≈ 1,200 |
| Instructions | 200 | ≈ 270 |
| Four worked examples | 500 | ≈ 670 |
| Input total | 3,200 | ≈ 4,300 |
| Output: JSON triage record | 120 | ≈ 160 |
Two things fall out of the table before any arithmetic. The customer's own text is about 3,300 of the 4,300 tokens, and nothing in the prompt changes that. And the output is a twenty-seventh of the input, so even at 4r the output is a small share of the cost; this is a reading workload, not a writing one.
One day, and its peak
1,200 claims a day:
input 1,200 × 4,300 ≈ 5.2M tokens/day
output 1,200 × 160 ≈ 0.19M tokens/day
The average rate is 1,200 ÷ 36,000 s (a ten-hour working day) ≈ 0.033 claims/s, and it is the wrong number to size against, for the reason back-of-the-envelope estimation gives: nothing runs at the average. Claims arrive 08:00–18:00 with 60% between 09:00 and 12:00, so the peak is
720 claims ÷ 3 h = 240 claims/h ≈ 0.067 claims/s
240 × 4,300 ≈ 1.0M input tokens/h ≈ 17k tokens/min
The 17k tokens a minute is the number to check against the hosted API's rate limit, which is quoted in tokens per minute, and against the quota on a listing in the customer's tenant. For a pilot it is comfortably small. It stops being small when the sponsor asks for the 40,000-claim backfill in week 3: 40,000 × 4,300 ≈ 172M tokens, which at a limit sized to the pilot's 17k a minute is seven days of continuous calls, or a request for a higher limit made a week before you need it.
Requests in flight
Little's Law from the system-design course gives concurrency as arrival rate × residence time, and it takes the mean residence time, not the p99. The p99 is useful only as an upper bound.
Assume the model answers in a mean of 8 s: reading 4,300 tokens and writing 160 at an assumed 50 tokens/s is about 3 s of decode plus the prefill and the round trip, and the p99 target from the definition of done is 20 s.
mean 0.067 /s × 8 s ≈ 0.5 requests in flight
p99 0.067 /s × 20 s ≈ 1.3 as an upper bound
One worker, with a queue that holds a few minutes of claims, is the whole fleet. The number is small enough that the temptation is to skip the calculation, and the reason not to is the backfill above: 40,000 claims that must finish by Monday is 40,000 ÷ 172,800 s ≈ 0.23/s, which at 8 s is two in flight, and at a 20 s tail during the vendor's busy hour is five. The pool size changes with the workload, not with the model, and a worker count written down in the scoping doc is the thing you point at when week 3 arrives.
Cost, as a multiple of r
The contract prices input at r per million tokens and output at 4r.
per day 5.2 × r + 0.19 × 4r ≈ 5.2r + 0.77r ≈ 6.0r
per claim 6.0r ÷ 1,200 ≈ 0.005r
The answer stops there. The customer has r in their contract and multiplies; an engineer who quotes a figure in currency has either guessed the rate or revealed it, and both are worse than 6r a day. What the multiple does show is the structure: cost is linear in volume, 87% of it is input, and 77% of the input is the customer's own documents. The levers on cost are therefore the prompt's 940 tokens (instructions and examples), about a fifth of the bill, and the choice of what to retrieve, about another quarter (1,200 tokens, ≈ 1.4r a day). Cutting the four examples to two saves about 335 tokens a claim, 8% of the input or about 7% of the cost (0.4r a day), and the only reason to do it is if the 200-case eval shows they add nothing; cost cannot justify it on its own at 0.005r a claim.
Two cases change the structure. A long-document workload (a 40-page contract at about 12,000 words is 16,000 tokens of input) is still input-dominated and scales the same way. A drafting workload (a 900-word customer letter from a 300-word brief) flips: 1,200 output tokens at 4r against 400 of input is 4,800r-equivalents against 400, and the output price is 92% of the bill. State which kind you have before the prompt work starts, because the lever is different.
Writing it down
The sizing paragraph in the scoping doc has six lines, each with its assumption attached:
- Tokens per document: 4,300 in, 160 out, at 0.75 words/token (measure on day 1).
- Volume: 1,200 a day, 5.2M input tokens, 0.19M output.
- Peak: 240 an hour, 17k tokens a minute, against the rate limit of the placement chosen in where the model sits.
- Concurrency: under 1 at the mean, 1.3 at the p99 bound; one worker, a queue of ten minutes.
- Cost: 6.0r a day, 0.005r a claim, 87% input.
- The event that changes any of them: a backfill, a second claims type, a longer policy section.
A sponsor who reads those six lines can sign the definition of done's cost line without a meeting.
In an interview
The estimate is being graded on the order and the assumptions, not on the final figure. Say the words-to-tokens ratio as an assumption, convert one document before the day, find the peak before the concurrency, and give cost as a multiple of r with the share that is input. Then say what the estimate cannot see: the constraints that decide the pilot are agreement and latency, and the cost number's job is to show that nothing in the budget is going to stop the pilot.
The mistake that loses points is quoting a per-token price from memory and producing an absolute figure. It is unverifiable, it dates the answer, and the customer can do the multiplication themselves in one step. Close behind: sizing concurrency from the daily average (0.033/s × 8 s is 0.27, which looks like "nothing" and misses the backfill), and using p99 as the residence time in Little's Law, which overstates the fleet by the ratio of tail to mean and leads to a pool nobody needs.
Check yourself
1. The policy section retrieved per claim grows from 900 words to 2,400 because the customer wants the full endorsement schedule included. What happens to the daily cost and to the lever you had?
Input per claim goes from 4,300 to about 6,300 tokens (2,400 ÷ 0.75 ≈ 3,200 against 1,200 before). Daily input is 1,200 × 6,300 ≈ 7.6M, cost ≈ 7.6r + 0.77r ≈ 8.3r a day, or 0.007r a claim, a 40% rise and over the 0.005r line. The prompt's 940 tokens are now 15% of the input instead of 22%, so the few-shot lever is smaller still. The real lever is retrieval: if the eval shows the schedule only matters for 1 claim in 5, retrieve it conditionally and the cost falls to about 6.4r, 0.0053r a claim, still over the line. The schedule is the customer's request, so moving the cost line is the sponsor's decision, made with this arithmetic in front of them.
2. A second customer has 300 claims a day but each claim carries a 30-page medical report of about 9,000 words. Size one day and say whether the shape of the cost changes.
One claim: 9,000 ÷ 0.75 ≈ 12,000 tokens of report, plus the same 2,100 of policy, instructions and examples, ≈ 14,000 in; output stays ≈ 160. Per day: 300 × 14,000 ≈ 4.2M input, 0.05M output, cost ≈ 4.2r + 0.2r ≈ 4.4r, or 0.015r a claim. The shape does not change: still input-dominated, now 95% input, and the customer's document is 86% of it. The lever that appears is summarising the report once and reusing the summary across the claim's later steps, which is a retrieval decision, not a prompt one.
3. The sponsor wants the 40,000-claim backfill done over one weekend on a hosted API limited to 20k input tokens a minute. Is it possible, and what do you ask for?
40,000 × 4,300 ≈ 172M tokens; at 20k a minute that is 8,600 minutes, about 6 days, so a weekend is not possible at the current limit. Ask the vendor for a temporary limit of 100k a minute, which brings it to about 29 hours, or agree with the sponsor to run it over a week in the background at the normal limit. Concurrency at 100k a minute is about 0.39 claims/s × 8 s ≈ 3 workers, so the fleet needs changing too, which is why the request is made a week ahead and not on Friday.