Latency numbers that change designs
Attach a real number to every hop in a design and budget a p99 target across sequential and parallel calls, so arithmetic picks the topology.
A main memory read takes 100 ns. A round trip from Mumbai to Virginia takes about 200 ms — two million times longer. Most architectural arguments worth having are really arguments about which of those numbers a request pays, and how many times. Memorise the ladder to one significant figure; the consequence attached to each rung is the part that gets scored.
The ladder
| Operation | Time | Scaled up 1 billion times |
|---|---|---|
| L1 cache reference | 1 ns | 1 second |
| Main memory reference | 100 ns | 100 seconds |
| SSD random read | 100 µs | 1.2 days |
| Round trip within a datacenter | 0.5 ms | 6 days |
| Disk seek (spinning) | 10 ms | 4 months |
| Round trip India → US East | ~200 ms | 6 years |
These are orders of magnitude; a given disk or path can be 2–3x off either way.
Memory is 1,000x faster than SSD, which is the entire case for caching
100 ns against 100 µs is a factor of 1,000: 50 small lookups cost 50 × 100 µs = 5 ms from SSD and 50 × 100 ns = 5 µs from memory.
The honest version includes the network. A Redis hit in the same datacenter costs one round trip, 0.5 ms, plus a 100 ns lookup — the lookup is 0.02% of the cost. So a remote cache is not 1,000x faster than a database; it is roughly 0.5 ms against 3–5 ms, a 6–10x win earned by skipping query planning, locking and disk contention. The full 1,000x needs an in-process cache, with no round trip at all.
That reframes cache tuning: collapsing 50 sequential GET calls into one MGET takes 25 ms to 0.5 ms, beating any hit-rate improvement available. Round trips, not lookups, are the unit of cost — see caching for eviction and invalidation.
A cross-region round trip is 400x a same-datacenter one
0.5 ms against 200 ms is a factor of 400, and it is physics rather than engineering. Mumbai to northern Virginia is roughly 13,000 km, and light in fiber moves at about 200,000 km/s: one way is 13,000 ÷ 200,000 = 65 ms, a round trip 130 ms. Real paths detour and switches queue, landing the measured figure near 200 ms.
Connection setup is itself round trips: TCP is 1 RTT and TLS 1.3 is 1 RTT, so 400 ms passes before a cold cross-region request is even sent. Reuse connections and terminate TLS near the user — much of what a CDN buys, separate from caching bytes at the edge.
Calling the API in the other region costs 200 ms, and three chained calls cost 600 ms — more than most whole p99 budgets. Replicate into the user's region and read locally instead, accepting that a follower lags its leader: a user who writes then immediately reads can be served the pre-write value and watch their own update vanish. See consistency models.
A disk seek costs 20 datacenter round trips
10 ms ÷ 0.5 ms = 20. An index lookup touching 4 pages at random costs 4 × 10 ms = 40 ms on spinning disk and 4 × 100 µs = 0.4 ms on SSD. That gap is why storage engines are built around access pattern: append-only commit logs, LSM trees, columnar layouts that read one contiguous run.
Take 1 GB. Sequentially, a fraction of a second. As a million random 1 KB reads: 1,000,000 × 100 µs = 100 seconds on SSD, or 1,000,000 × 10 ms ≈ 3 hours on spinning disk. Same bytes, three orders of magnitude apart.
Budgeting a p99 target
Work backwards from the requirement, not forwards from the components. Target p99 is 200 ms server-side, and the request touches 5 internal services, each called in turn: 200 ms ÷ 5 = 40 ms per call.
That 40 ms is not 40 ms of work. Per-hop overhead — a 0.5 ms round trip, serialization at both ends, internal TLS — runs 2–3 ms, so five hops burn 10–15 ms on transport alone, leaving about 37 ms per service.
Then the composition trap. Five calls each at p99 = 40 ms do not compose into a chain at p99 = 200 ms; the chain is fast only when every call is fast. For independent calls, the probability all five land inside their own p99 is 0.99^5 ≈ 0.951 — about 4.9% of requests carry a call in its tail, so the chain's 99th percentile sits nearer each call's p99.8. Budget each dependency at p99 ≈ 25 ms and keep the rest as headroom. Tail latency is a property of the chain, so measure it end to end instead of assembling per-service percentiles — see monitoring and alerting.
Fan out instead of chaining
Independent calls belong in parallel: serial cost is the sum, parallel cost is the maximum. Three calls at p99 40 ms cost 40 + 40 + 40 = 120 ms serially; in parallel, max(40, 40, 40) ≈ 40 ms plus a few milliseconds to fan out and merge — call it 45 ms.
Fan-out trades a latency problem for a tail problem: the response waits for the slowest branch, and the probability all three finish inside 40 ms is 0.99^3 ≈ 0.970, so ~3% of requests exceed a single call's p99. The fix is a per-branch timeout and a degraded response — the feed without its recommendation strip if ranking misses 50 ms — not a longer timeout.
Where calls genuinely depend on each other, the levers are depth and colocation: merge two services that always call each other, denormalise away the second lookup, or precompute off the request path. Every synchronous level adds latency, a failure mode and a tail.
In an interview
What is being tested: whether you can attach a number to a box on the whiteboard, and whether that number then changes the drawing.
Name the rung, then the consequence: "that is a same-datacenter round trip, 0.5 ms, so twenty of them is 10 ms and I can afford it; this one is cross-region, 200 ms, so I can afford zero on the read path and will replicate." Then budget out loud: "200 ms over four sequential hops is 50 ms each including transport, and ranking measures 80 ms, so it moves off the synchronous path."
The mistakes that lose points:
- Reciting the ladder with no design consequence attached.
- Treating an extra service hop as free. Each is +0.5 ms at best, plus serialization and a new independent tail.
- Adding averages. "Each call averages 20 ms, so five is 100 ms" is the mean; users feel p99, and tails compound.
- Answering a 200 ms cross-region problem with a cache in the origin region. A cache in us-east-1 does nothing for a user in Mumbai.
Check yourself
1. A handler makes 6 sequential Redis GET calls in the same datacenter, then a Postgres query reading 3 pages from SSD. Estimate server-side latency and name the biggest single fix.
6 × 0.5 ms = 3 ms of Redis round trips, plus a 0.5 ms round trip to Postgres and 3 × 100 µs = 0.3 ms of SSD reads: roughly 3.8 ms, of which 3.5 ms is network. One
MGETinstead of six gives 0.5 ms + 0.8 ms ≈ 1.3 ms — a 3x cut, storage untouched.
2. Your origin runs in us-east-1, 30% of users are in India, and a page makes 5 sequential API calls from the browser against a 300 ms p99 target. Does it fit?
No. Each browser-to-origin round trip is ~200 ms, so 5 sequential calls cost 1,000 ms of pure network — 3x over budget before any server work. Aggregate the five into one (200 ms), then move that aggregation to an edge or a Mumbai region so the round trip drops to roughly 20 ms. A CDN covers static assets; the API needs a nearer origin or replica.
3. Your p99 budget is 200 ms and the request needs 5 internal calls, each measured at p99 = 35 ms. Ship it?
Not on those numbers. Serially it is 175 ms, apparently 25 ms of margin, but the probability all five land inside their own p99 is 0.99^5 ≈ 0.951 — ~5% of requests carry a tail call, so the chain p99 sits well above 175 ms. Parallelise the independent calls (max ≈ 45 ms) or re-budget each dependency to p99 ≈ 25 ms, then measure end to end.