Tail latency: why p99 is the number that matters

A service whose dashboard reads 19 ms mean can still make one page load in five feel broken. The mean averages over requests; a user feels the slowest request their page needed. Those two numbers diverge the moment the distribution has a tail, and every real one does.

The mean hides the tail

Take a service where 99 of every 100 requests return in 10 ms and the hundredth takes 900 ms — a GC pause, a cache miss that fell to disk, a request queued behind something large.

mean = 0.99 x 10 ms + 0.01 x 900 ms
     = 9.9 + 9.0
     = 18.9 ms

The dashboard reads 19 ms, p50 is 10 ms, p99 is 900 ms. All three are correct; one predicts what a user feels.

Put that service behind a page: one feed screen takes 20 API calls, each with a 1% chance of the slow path:

P(no slow call)       = 0.99^20 = 0.818
P(at least one slow)  = 1 - 0.818 = 0.182

About 18% of page loads contain a 900 ms call: the service's p99 is the page's p82. Nobody in that 18% cares that the mean is 19 ms.

Fan-out turns p99 into p50

Sequential requests are the gentle case. Parallel fan-out is the hard one: a page assembler calls 50 shards at once and cannot render until all return. Page latency is not the average of 50 samples, it is the maximum.

The page's median sits at the per-call quantile q where all 50 calls land below it half the time:

q^50 = 0.5
q    = 0.5^(1/50) = 0.986

The page's p50 is the backend's p98.6: half of all page loads are gated by a call the backend team files under rare outlier.

Fan-out	Backend quantile setting the page median	P(page hits the 900 ms path)
1	p50	1%
10	p93.3	1 − 0.99^10 = 9.6%
50	p98.6	1 − 0.99^50 = 39.5%
100	p99.3	63.4%

Fan-out amplification: one slow shard in fifty sets the page latency100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

Large fan-out systems obsess over the tail for this reason: at a fan-out of 50, cutting the backend p99 from 900 ms to 90 ms improves the median page load, while halving its p50 changes nothing a user perceives.

Where the tail comes from

Garbage collection. A 200 ms stop-the-world pause stalls every request in flight. By Little's Law a server at 100 QPS holds 100 x 0.2 = 20 requests, so each pause catches 20. One pause per 30 s is 20 victims per 3,000 requests: a 0.67% slow rate, exactly where p99 lives.

Queueing behind a slow request. Expected queue wait is ρ/(1−ρ) service times. At 95% utilisation that is 0.95/0.05 = 19x — 950 ms of waiting on a 50 ms request before work starts. At 80% it is 0.8/0.2 = 4x, or 200 ms. Utilisation, not code, is the dominant term.

Cold caches. A hit is a sub-millisecond round trip inside the datacenter; a miss is a ~100 µs SSD read plus a query costing 100x that. A 98% hit rate is a 2% slow rate — see caching.

Retries. A retry after a 1 s timeout guarantees the caller sees 1 s plus the second attempt, and it adds load exactly when the dependency is struggling — which is why retries belong behind a circuit breaker and a budget, not in a loop.

Noisy neighbours and skewed shards. A co-tenant saturating shared disk, or one shard holding a celebrity key while 49 idle, produces a slow minority that average-case tuning cannot touch — see database scaling for how a shard key creates it.

Remedies, cheapest first

Hedged requests. Send the request; if no answer by the p95 mark, send a second copy to another replica and take whichever returns first. The cost is bounded by construction: only 5% of requests are still outstanding at their own p95. Google's published measurement on a BigTable-backed service cut the 99.9th percentile from 1,800 ms to 74 ms for about 2% extra requests. A hedge is a deliberate duplicate, so the operation must be idempotent: safe for reads, unsafe for writes unless the handler deduplicates.

Timeouts set from the distribution. A timeout is a claim about the tail, not a round number someone typed. With p99 at 900 ms, 1 s cuts off the pathological cases; 30 s lets a hung dependency hold each thread for 30 s.

Bounded queues and load shedding. Depth should be at most timeout x drain rate: 500 QPS with a 1 s timeout gives 500 slots. Work accepted beyond that expires before a worker reaches it, so shed it with an immediate 503 — a fast rejection costs one client a retry decision, a slow acceptance costs everyone latency. An unbounded queue turns a capacity problem into a latency problem plus a memory leak, and the same arithmetic governs a message queue backlog.

Reduce fan-out. Batching 50 shard calls into 10 moves the page median from the backend's p98.6 to its p93.3 and the slow-path chance from 39.5% to 9.6%. The most durable fix, and the most expensive.

Provision for the tail, not the mean

Little's Law says concurrency = arrival rate x latency. At 500 QPS on our example service:

fast path: 495 QPS x 0.010 s = 4.95 concurrent
slow path:   5 QPS x 0.900 s = 4.50 concurrent
total                        = 9.45 concurrent

One percent of requests hold 48% of the concurrency. Size the pool from p50 and you get 500 x 0.010 = 5 workers, about half of what the service needs. The error also compounds: if the slow fraction rises to 5%, demand becomes 25 x 0.9 + 475 x 0.01 = 27 concurrent, nearly 3x the pool. The pool exhausts, requests queue, queueing lengthens the tail, and a longer tail raises the slow fraction again.

In an interview

The interviewer is testing whether you know that a distribution is not a number. Quote the percentile whenever you quote a latency — "p99 of 200 ms at 1,000 QPS", never "200 ms" — and when the design fans out, state the amplification before being asked: "we call 40 services in parallel, so the page median tracks the backend p98; we need hedging or a smaller fan-out."

Name what you will measure: percentiles computed server-side per endpoint and per shard, since averaging p99 across hosts produces a number describing no request. Monitoring and alerting enforces that.

The mistake that loses points is offering a mean, or an SLA of "sub-100 ms average", as evidence the design is fast, then not noticing when the same design fans out. Its second form is proposing retries as a tail remedy with no budget and no breaker, which turns a slow dependency into a dead one.

Check yourself

1. A backend has a p99 of 200 ms and a p50 of 8 ms. A page issues 30 parallel calls and waits for all. What fraction of page loads includes at least one 200 ms call, and which backend percentile sets the page median?

1 − 0.99^30 = 1 − 0.740 = 26% include a 200 ms call, and 0.5^(1/30) = 0.977 puts the page median at the backend's p97.7. The 8 ms p50 describes almost nothing the user experiences.

2. You have 5% spare capacity and two proposals: hedge at p95, or cut fan-out from 40 to 10. Which ships first, and what does each buy?

Hedging fits the 5% budget almost exactly — only requests past their own p95 get a duplicate — and ships in days. Cutting fan-out from 40 to 10 moves the page median from p98.3 to p93.3 and the slow-path chance from 1 − 0.99^40 = 33% to 9.6%, but needs a batching API and a new assembler. Hedge now on idempotent reads; re-shape the fan-out next quarter.

3. A worker pool runs at 95% utilisation with 50 ms of service time. A colleague proposes optimising the handler to 40 ms. Estimate the effect, and propose a better lever.

Queue wait dominates: at ρ = 0.95 it is 19 x 50 = 950 ms, so a request takes ~1,000 ms. Cutting service time to 40 ms drops utilisation to 0.76 and the wait to 0.76/0.24 = 3.2 x 40 = 127 ms — a real win, but it came from the utilisation, not the 10 ms. Adding 25% more workers buys the same drop with no code change.