The non-deterministic UI
Place a 2.1 s median and 6 s p95 model call against the 0.1 s, 1 s and 10 s limits, and decide what the screen shows at each second of the wait.
On this page
The same drafting call takes 2.1 s at the median and 6 s at the p95, and run twice on one ticket it returns two different drafts. The screen cannot make the call faster, but it decides which number the agent feels, and with streaming that number is the first token at about half a second rather than the last at six.
Three limits that describe people
Jakob Nielsen's Usability Engineering (1993), building on Robert Miller's 1968 study of response times, gives three limits that have not moved in fifty years because they describe attention, not hardware.
- 0.1 s feels instantaneous.
- 1 s keeps the flow of thought unbroken; the user notices the delay but stays on the task.
- 10 s is the limit of attention; past it the user turns to something else, and the screen owes them progress and a way to stop.
For an agent on a 2-minute reply cycle: within 0.1 s of a click something on screen changes; within 1 s there is something to read; by 10 s the agent has the composer back, whatever the model is doing.
Where a model call lands
Reading an architecture added up the path: gateway about 1 ms, retrieval about 80 ms, model 2,100 ms at p50 and 6,000 ms at p95. Both sit between 1 s and 10 s, the worst band: long enough to break the flow, short enough that the agent waits. The p95 is the one to design for, because the tail is what users feel: at one tenant's 48,000 drafts a day, at least 2,400 take 6 s or longer.
The naive design blocks: a spinner until the whole draft is ready. At the median, the tenant's agents watch 48,000 × 2.1 s = 100,800 s of spinner a day, about 28 agent-hours. A smaller model decodes faster, as model selection sets out, and a 150-token cap halves the decode, but both take weeks and an eval run. The lever that takes days is changing what the agent waits for.
Streaming changes which number the agent feels
A call has two phases: prefill reads the prompt and produces the first token, and decode produces the rest one at a time (latency budgets and streaming). Streaming forwards each token as it is made. The first leaves the model at about 400 ms and is on screen at about 500 ms; the 300-token draft follows over the next 1.7 s.
Does the agent then wait on the stream? Assume a reading speed of about 250 words a minute, roughly 4 words a second. The draft is about 230 words; at the median it arrives at about 135 words a second, and at the p95, decoding over about 5 s, at 45 words a second, still ten times reading speed. From the first line, the text is ahead of the eye.
So the felt wait is the first token. Assume it arrives by 0.9 s at the p95 (read it off the trace): streaming turns a 6 s wait into a 1 s one without touching the model. At the median the tenant saves (2.1 − 0.5) s × 48,000 = 76,800 s, about 21 agent-hours a day, a minute per agent. The larger gain: attention never leaves the ticket.
Insert and Send still wait for the last token, so the last-token time matters, but only to the button.
What the screen shows, second by second
0 to 0.1 s. The suggestion area appears with a skeleton, grey lines about the length of a typical draft.
To about 0.5 s. Loading. Not a spinner, which says "wait", where a skeleton says "look here". Not a percentage bar either: the model does not know how long its answer is until it stops, so any percentage is invented.
From about 0.5 s. Streaming. Text replaces the skeleton line by line; Insert is visible but disabled, with Stop beside the text. The agent can type in the composer at any time.
2.1 s at the median. Done. Insert goes live.
4 s. Slow: about twice the median, so this is not a normal call. "Still writing" appears with Cancel, and the composer stays editable; if the agent starts typing, the draft collapses. This is also where the done line's "p95 under 4 s" comes from: under 4 s at the p95, at most one draft in twenty shows the slow line. At today's 6 s p95 it is more than 2,400 a day, which is why the PRD treats p95 as a requirement.
10 s. Give up, at the attention limit. Stop the stream; if what arrived ends on a full sentence, keep it as a partial draft with Continue and Regenerate, otherwise show the error reason. No automatic retry: the agent has moved on. The timeout follows the output cap, as in retries, timeouts and rate limits: 300 tokens at the p95's 60 a second is 5 s of decode, so 10 s is twice the p95 decode.
The mistake: testing at the median
Tested at 9 am on office wifi, drafts land in about 2 s and a spinner looks fine. At 4 pm under load one draft in twenty takes 6 s; agents start typing their own reply at 3 s, the draft lands on top of it, and the pilot's feedback reads "the AI overwrites my reply". The design was tested at the p50 and shipped to the p95. The fix: design for the p95 and the timeout, never let a late draft overwrite agent input, and inject 6 s and 12 s delays before the pilot.
Same ticket, two different drafts
Sampling means the same prompt can return a different draft each time, and temperature 0 narrows that without guaranteeing it (sampling and determinism). Three decisions follow.
Regenerate replaces the draft and keeps the previous one a step back. Two drafts, not a list.
Regenerate skips any per-ticket draft cache; a cache keyed on the ticket returns the same text, and the button looks broken.
No three alternatives side by side. At an assumed $0.009 a draft, three per ticket cost 48,000 × 3 × $0.009 = $1,296 a day against $432, about $25,900 a month more, and the two extra 230-word drafts take nearly 2 minutes to read at 4 words a second, more than the minute per reply the feature saves.
In an interview
"The model takes 2 to 6 seconds. How do you design the screen?" tests whether you know the three limits and which latency the user feels. Answer as a sequence: "Skeleton inside 0.1 s. First token at about 400 ms, so we stream and stay under 1 s at both percentiles. Insert waits for the last token. Slow line with Cancel at 4 s, about twice the median. Timeout at 10 s. A late draft never overwrites what the agent typed." Then give the p95 count, at least 2,400 a day.
The mistake that loses points is designing for the median: quoting 2.1 s and stopping, or proposing a progress bar for an answer whose length nobody knows. The second is treating regenerate as free; it is a new call, a new cost and a new answer.
Check yourself
1. After a model update the p95 first token rises from 0.9 s to 2.5 s; the p95 last token stays at 6 s. Decide what changes.
One draft in twenty now shows only the skeleton for 2.5 s or longer, past the 1 s limit, at least 2,400 times a day; the 4 s slow line is unaffected. The cheapest fix hides the regression: the draft already starts when the ticket opens, so 2.5 s overlaps the agent reading the ticket. Then log it as a regression on the update.
2. Engineering offers two sprints to cut the median last token from 2.1 s to 1.5 s, or three days to ship streaming. Which first?
Streaming: the felt wait goes from 2.1 s to about 0.5 s at the median and from 6 s to about 0.9 s at the p95. The optimisation leaves the median over the 1 s limit without streaming and says nothing about the p95. After streaming, the two sprints are worth it only if they bring the p95 last token under 4 s, which is what moves the slow-line count.
3. The timeout is set at 30 s "to be safe". The service peaks at 10 drafts a second. Estimate the cost when the vendor stalls, and decide.
By Little's law, 10 a second × 30 s = 300 open streams during a stall against 100 at 10 s, three times the connections for drafts nobody is waiting for: the agent passed the 10 s attention limit and typed the reply. Set 10 s.