Latency budgets and streaming
Split a model call into prefill and decode, set a p95 budget for a 4-round-trip agent, and decide when to stream and when to move the work off the user path.
On this page
A model call has two phases with different clocks. Prefill reads the whole input before the first output token appears, and its time scales with input length. Decode then produces tokens one at a time, and its time scales with output length — at the course's decode speed of 50 tokens/s, from the four numbers, an 800-token answer is 16 seconds whatever the input was. Most latency surprises come from budgeting against the input, when the output is the term that dominates.
Two phases, two clocks
Time to first token is the prefill phase plus queueing at the vendor. Prefill runs in parallel across the input, so it is much faster per token than decode; for a prompt of tens of thousands of tokens it is typically a second or two, and a cache hit on the prefix (see prompt caching) shortens it further, since the cached part is not reprocessed. Measure yours: the four numbers assumes about a second for a few thousand tokens of input, and it is the assumption most worth replacing with a measurement.
Decode is the predictable half. Tokens arrive at roughly the decode speed, so output time is output_tokens ÷ 50 seconds:
60 output tokens (a tool call) ≈ 1.2 s
300 output tokens (a short answer) ≈ 6 s
800 output tokens (a page of prose) ≈ 16 s
The mistake is to look at a 30k-token prompt, worry about prefill, and ignore that the 800-token answer the prompt asks for takes 16 seconds on its own. Shortening the answer is the latency lever, the way shortening the input was the cost lever in where the money goes.
Budgets are p95, not means
Output length varies per request, so latency varies with it — a classifier that answers in 20 tokens most of the time and 400 when it decides to explain itself has a mean near 1 second and a tail near 9. Add vendor queueing, which comes in bursts, and retries, which double a request's time, and the mean describes no request a user actually saw. Tail latency makes the general case; for model calls the output-length variance makes the tail wider than for most services.
A budget is therefore a statement about a percentile on a path: "p95 of the user-visible path under 10 s". Write it that way, measure it that way in the trace, and gate releases on it beside accuracy, as regression gates does. Two controls make it holdable: cap max_tokens so the long tail of output cannot exist, and cap the turn count in the loop so the tail of turns cannot either.
Where the seconds go in a 4-round-trip agent
An agent that answers a question by calling three tools makes four model calls. The trace below assumes a 6k prefix cached after the first call, 60-token tool calls, tool latencies of half a second to a second, and a 300-token final answer.
Totals, by kind:
prefill (4 calls, 3 cached) ≈ 2.0 s
decode of 3 tool calls ≈ 3.6 s
3 tool executions ≈ 1.9 s
decode of the 300-token answer ≈ 6.0 s
≈ 13.5 s, plus queueing → 14.7 s
Against a 10 s budget this fails, and the fix is not a faster vendor. Three options, in order of size: cut the answer to 150 tokens (saves 3 s); issue the three tool calls in one message where they are independent — the 180 tokens of calls still decode, but two queue waits and two prefills disappear and the tools run side by side, 0.8 s instead of 1.9 s (saves about 2.7 s); and stream the final answer so the user sees text at 8.7 s rather than 14.7 s. The first two together finish at about 9 s. Streaming alone meets the budget only if the budget was written for the first visible word rather than the last, so settle which before choosing.
Stream, or run it in the background
Streaming does not shorten a call; it moves the moment the user sees something from the end of decode to its start. The decision rule has two inputs: is a person waiting, and is the output long enough to notice.
- Person waiting, output over ~100 tokens (2 s of decode): stream. The first token arrives after prefill, and reading speed is below 50 tokens/s, so the user never catches up with the model.
- Person waiting, output under ~100 tokens: don't bother. A classification or a tool call arrives in a second or two whole, and streaming adds client complexity for nothing.
- Nobody waiting: run it in the background, write the result where it is needed, and notify. The contract summary that a reviewer opens up to 30 minutes after upload should never hold an HTTP connection open.
Long outputs need streaming regardless of the user. An 8,000-token document at 50 tokens/s is 160 seconds of decode, past most HTTP and gateway timeouts; a streamed connection stays alive because bytes keep flowing. The timeout side of that is in retries, timeouts and rate limits.
Streaming also changes what you can check. The four checks from reading the response — stop reason, schema, invariants, provenance — run on the complete message, so a streamed JSON object is validated at the end, not shown to the user as it arrives. Stream prose; buffer structure.
Cutting the budget
Levers, largest first:
- Shorten the output. Every 50 tokens removed is a second. Ask for the fields, not the essay; cap
max_tokensat what the consumer reads. - Collapse rounds. Independent tool calls in one message, as tool use and the tool-call loop shows; each round removed saves a prefill and a queue wait, and the tools run side by side instead of one after another. The calls' own tokens still decode.
- Cache the prefix. Cuts prefill on every call after the first.
- Route the intermediate turns. A smaller tier decodes faster; the tool-calling turns rarely need the large one. The eval decides, per model selection and routing.
- Move work off the path. Precompute what can be precomputed — the summary of the ticket thread at ticket update, not at agent open.
Throughput, as opposed to a single request's latency, is a different budget, and the operator's view of it is in throughput and latency for automations.
In an interview
"Can this answer in three seconds?" is the question, and what is tested is whether you split the call into prefill and decode before answering. Say the decode arithmetic first: "300 tokens at 50 a second is 6 s before anything else, so no — unless we stream, shorten, or precompute". Then name the percentile: "p95, because output length varies and the mean hides the long answers". Then the levers in order of seconds saved.
The mistake that loses points is quoting a mean, or budgeting on input length when the output decode is the dominant term. A candidate who says "the prompt is small so it'll be fast" and then asks for a 500-word answer has not done the division.
Check yourself
1. A feature returns a 120-word answer to a user who is watching. Stream it?
120 words × 1.3 ≈ 160 tokens, about 3.2 s of decode after a 1 s first token. Four seconds of blank screen is long enough to notice, so stream: the first words land at about 1 s and the user reads behind the model. If the answer were 30 words, 40 tokens, under a second of decode, deliver it whole.
2. An agent's p95 is 22 s on a 12 s budget. The trace shows 5 rounds, 60-token tool calls, 0.8 s tools, a cached prefix, and a 500-token answer. Which lever first?
Decode of the answer is 10 s of the 22; the five rounds are about 5 × (0.5 + 1.2 + 0.8) ≈ 12.5 s. Cut the answer to 200 tokens: saves 6 s, p95 ≈ 16 s. Then merge the independent tool calls into one or two rounds: saves another 5 s or so, p95 ≈ 11 s. Streaming on top makes the perceived wait about 7 s. A faster model for the tool-calling turns is fourth, and needs an eval run first.
3. A document generator writes 6,000 tokens per run and the caller is a nightly job. Stream, background, or batch?
6,000 ÷ 50 = 120 s of decode: too long to hold a plain request open, and nobody is watching. Batch it — the job has hours — and if the batch endpoint is unavailable for the feature, run it as a background job with a streamed connection so the 120 s does not trip a gateway timeout. Streaming to a user is the wrong frame when there is no user.