Caches and the cost of a miss
Size a cache from its hit rate, price the miss path in milliseconds, and stop a cold restart from taking the database down with it.
A cache stores data so that future requests for it can be served faster. The value is either the result of an earlier computation or a copy of something that lives elsewhere, usually in a database. Caching works because the layers underneath a system differ by orders of magnitude: a main memory reference costs about 100 ns, an SSD random read about 100 µs, a spinning disk seek about 10 ms. Moving a copy up one rung buys a factor of a thousand.
The read path
The common arrangement is cache-aside: the application, not the datastore, owns lookups.
The application checks the cache first. If the value is there — a cache hit — it is returned immediately. If not — a cache miss — the application reads the original source, stores a copy, and returns the value. Later requests for that key are served from the cache until the entry is evicted or invalidated.
What a hit rate is worth
Take a news site: 50 million daily actives, 20 requests each. That is 50M × 20 = 1 billion requests/day ÷ 86,400 s ≈ 12,000 QPS average, and peaks run 2–5× average, so assume 36,000 QPS at peak.
A single commodity Postgres box handles around 5,000 simple queries per second, so peak is seven times its capacity and the cache has to absorb at least (36,000 − 5,000) ÷ 36,000 ≈ 86% of reads before the origin is viable at all. At a 95% hit rate the origin sees 36,000 × 0.05 = 1,800 QPS and one box is comfortable. At 80% it sees 7,200 QPS and we need read replicas, whose lag can make a user's own edit vanish from their next read, or a shard. Fifteen points of hit rate is the difference between one database and three.
Latency moves the same way. A hit costs one datacenter round trip, 0.5 ms, since the memory lookup inside Redis is noise beside the network. A miss costs that round trip, the origin query and the write-back — round it to 6 ms. Mean latency at a 95% hit rate is 0.95 × 0.5 + 0.05 × 6 ≈ 0.8 ms.
That mean is misleading. With one miss in twenty, every request above the 95th percentile is a miss, so p99 is the miss path: about 6 ms, eight times the mean. To pull p99 onto the hit path you need a hit rate above 99%, not above 95%.
Sizing: why a small cache buys a large hit rate
Access is skewed, so you cache the head of the corpus, not the corpus. Two million articles at 1 KB of summary each is 2 GB, but if 95% of reads land on the 50,000 most recent and most shared, that hot set is 50,000 × 1 KB = 50 MB. Caching 2.5% of the data serves 95% of the traffic and fits in RAM anywhere. Measure the skew before buying memory.
The failure mode: a cold cache
The Redis node restarts at peak. Hit rate drops to zero and 36,000 QPS lands on a database provisioned for 1,800. Queries queue, and a 5 ms query now takes 200 ms. By Little's Law (concurrency = arrival rate × latency), in-flight work goes from 12,000 × 0.0008 ≈ 10 requests to 12,000 × 0.2 = 2,400. A pool of 200 connections cannot hold that: requests queue, time out, and are retried, adding load to a system already over capacity. The cache did not slow down, it disappeared, and nothing was designed to run without it.
Three cheap mitigations:
- Request coalescing (single flight). On a miss, one request per key goes to the origin and the rest wait on its result, turning 36,000 simultaneous front-page misses into one query.
- TTL jitter. Expire at 60 s plus a random 0–15 s. Without jitter, keys written together expire together and stampede once a minute.
- Warm before serving. Populate a restarted node before adding it back to the load balancer pool.
Eviction and invalidation
Eviction policies decide what to drop when the cache is full. LRU evicts the entry unused for the longest — the default, and right when access is recency-skewed. LFU evicts the least frequently used, protecting a stable hot set from a one-off scan. FIFO evicts in insertion order and ignores access entirely: cheapest to implement, weakest in effect.
Invalidation decides when a value stops being true. A TTL bounds staleness with no coordination: 60 seconds means readers can see 60-second-old data and you accept that. Invalidating on write is exact but harder, and the verb matters. Delete the key rather than overwrite it: two writers can be reordered between database and cache and strand a stale value, while a delete sends the next reader back to the source of truth. See consistency models for what stale data costs.
Write policies sit alongside this. Write-through writes to cache and store together: no stale row, slower write. Write-back writes to the cache and flushes later: fast, but an acknowledged write is lost if the node dies first. Write-around skips the cache and writes straight to storage, avoiding pollution from data nobody reads back, at the cost of a guaranteed first-read miss.
Where caches live
- Client-side. Browsers cache pages, images and assets, removing the request entirely.
- CDN. Servers caching static content near users — the only layer that helps with distance, since a round trip from India to US East is about 200 ms and no server-side cache removes it. See CDNs.
- Server-side, shared. Redis or Memcached, over the network at 0.5 ms.
- Server-side, local. An in-process map, at about 100 ns — five hundred times faster than Redis because there is no network hop.
Local caches look strictly better until you count copies. With 20 app servers an invalidation has to reach all 20, and until it does two users get different answers depending on which server the load balancer picked. Each server also sees 1/20 of the traffic, so each warms more slowly while duplicating the same 50 MB. We reject the local cache for anything a user can edit and keep it for feature flags and configuration, which tolerate seconds of skew — which is also why a cache makes an app server stateful in practice even when it looks stateless.
In an interview
The interviewer is testing whether you treat the cache as an optimisation with a failure mode or as free speed. Say the hit rate out loud, derive the origin load from it, and name the TTL. "I'll cache rendered summaries in Redis on a 60-second TTL with jitter; at a 95% hit rate that leaves about 1,800 QPS on the primary at peak, which one box handles" is a complete answer in one sentence.
The mistake that loses points is sizing the origin for the cached load and never asking what happens when the cache is empty. If the database survives only while the cache is warm, you have built a single point of failure and called it fast. State the cold-start behaviour, even if it is that you shed load and serve stale data. Two smaller ones: quoting a mean when the miss path owns p99, and saying you will invalidate on write without saying which write policy you mean. The caching deep dive goes further on strategy.
Check yourself
1. Your service takes 4,000 QPS at peak against one Postgres box rated at roughly 5,000 simple QPS. A product change triples traffic to 12,000 QPS. What hit rate do you need to stay on one box, and is that realistic?
The origin must stay under 5,000 QPS, so the cache absorbs
(12,000 − 5,000) ÷ 12,000 ≈ 59%. Leave headroom and target 80%, putting the origin at 2,400 QPS — achievable for skewed reads, not for per-user unique reads. A personalised feed has no shared hot set, so shard or add replicas instead.
2. A user edits their profile. Do you write the new value into the cache, or delete the key?
Delete it. Two concurrent edits can reach the database in one order and the cache in the other, leaving the older value in the cache with nothing but the TTL to clear it. Deleting forces the next read back to the source of truth, so the worst case is one extra miss rather than permanent staleness.