Cost, latency and routing · Latency and retries

Can the summariser answer in 3 seconds?

Work out whether a 250-word ticket summary can appear within 3 seconds of a click, and pick streaming, shortening or precomputing from the seconds.

estimateeasyExercise · 3 min · 28 of 77

On this page

Product has promised a summary on screen within three seconds of an agent opening a ticket. The engineer who inherits the promise has to find out whether the decode arithmetic allows it.

The situation

When a support agent opens a ticket, the UI calls the model with the ticket thread — on average 6,000 tokens — and asks for a 250-word summary before the agent reads the thread. The product requirement is that the summary is on screen within 3 seconds of the click, at p95. Tickets are opened 1,500 times a day; a ticket is updated by a customer or an agent about 4,000 times a day in total.

The current implementation makes the call on open, waits for the full response, then renders it. Nobody has measured it, but agents report "it takes ages".

Assume 1.3 tokens per English word, a decode speed of 50 tokens/s, and a measured time to first token of 1.2 s at p95 for a 6k-token prompt. Assume a background job costs the same tokens as a foreground call, and that reading from the ticket database takes milliseconds.

The question

Can the summary be on screen in 3 seconds as built — and if not, which of streaming, shortening the summary or precomputing it meets the requirement, with the seconds shown for each?

A strong answer · Rubric · Where answers go wrong

Write your answer first, then open

A strong answer

As built: 250 words × 1.3 ≈ 325 output tokens, decoded at 50 tokens/s is 6.5 s, plus 1.2 s to the first token: about 7.7 s at p95. Not 3 s, and no vendor is going to decode 3.6× faster next quarter.

Streaming: the first words appear at 1.2 s and the last at 7.7 s. That satisfies "something on screen in 3 seconds", which is what agents feel, but not "the summary in 3 seconds", which is what was promised; the agent is still reading a half-written summary at the 3 s mark.

Shortening: a 3 s budget minus 1.2 s of first-token leaves 1.8 s of decode, which is 90 tokens, about 70 words. A 70-word headline is useful; it is not the 250-word summary product asked for, and it only just fits.

Precomputing: summarise the ticket whenever it is updated, in the background, and store the result; the open then reads a row in milliseconds. Cost: 4,000 updates a day × 6k tokens is 24M input tokens a day against 9M for summarising on open — roughly 2.7× the tokens, and the summary is always current when the ticket is opened.

Decision: precompute on update, since the requirement is about the open and the open becomes a database read. Show a 70-word headline streamed on open only as a fallback for a ticket updated in the seconds before it was opened. The evidence that reverses it: if updates ran at twenty times opens rather than under three, precomputing would burn tokens on summaries nobody reads, and the answer would flip to streaming the full summary with the headline first — the trade in latency budgets and streaming.

Rubric

converts the 250 words to tokens and divides by the decode speed, adding time to first token
states the as-built p95 and that it fails the requirement
works each option in seconds: when streaming shows first and last words, how many words fit in 3 s
costs precomputing in tokens per day against summarising on open
picks precomputing and names the update-to-open ratio that would reverse the choice

Where answers go wrong

"The prompt is only 6k tokens, so it should be fast — probably a slow network." The output is the term that matters: 325 tokens is 6.5 s of decode before any network is involved. The candidate budgeted on input.

"Stream it; problem solved." Streaming moves the first word to 1.2 s and leaves the last at 7.7 s. If the promise was a summary in 3 s, streaming makes it feel better without meeting it, and the answer never checked which promise was made.

"Use a faster model." A smaller tier decodes faster, but the candidate gives no number, and even a doubling leaves 325 tokens at 3.25 s plus first-token — still over. The arithmetic rules the option out before an eval would.