Little's Law and why slow dependencies take systems down

A service accepting 500 requests per second, each held for 200 ms, has 100 requests inside it at every instant. Nobody measured that number — it is forced. Little's Law ties how many things are inside a system to how fast they arrive and how long each stays: the shortest path from a latency number to a capacity number. Two answers depend on it entirely — how large a pool should be, and why a dependency that gets slow is more dangerous than one that dies.

The law

L = λ × W

L  items inside the system   (requests in flight)
λ  arrival rate              (requests per second)
W  time each item stays      (seconds)

The units cancel: requests/second × seconds = requests. The only assumption is a stable window — arrivals roughly equal departures. Nothing is assumed about the latency distribution or scheduling policy, so it holds for any boundary you can draw: a thread pool, a queue, a datacentre.

Rearranged, it answers two more:

λ = L / W     ceiling of a fixed pool
W = L / λ     wait behind a backlog of L

Sizing a pool

Take a service at 500 QPS peak — 500 × 86,400 ≈ 43 million requests a day — with a p99 latency of 200 ms.

L = 500 /s × 0.2 s = 100 concurrent requests

A pool of 50 is therefore not "slightly undersized". It is a queue: its ceiling is λ = 50 / 0.2 s = 250 QPS, so 250 requests per second arrive with nowhere to go, and after 4 seconds a thousand are waiting 1000 / 250 = 4 s and climbing. No dashboard reports "pool too small"; it reports "latency up", the same fact in disguise.

Picking the number:

1. Use peak λ, which runs 2–5× average: a service averaging 150 QPS is sized for 500.
2. Use p99 W, not the mean — the tail occupies the slots. See monitoring and percentiles.
3. Multiply by about 2 for headroom: 100 → 200 workers.
4. Sanity-check downstream. 200 workers means up to 200 open database connections, and a commodity Postgres box handles roughly 5,000 simple QPS. A pool larger than what the dependency absorbs moves the queue rather than removing it.
5. Bound the wait queue: depth 50 in front of a 500 QPS pool adds at most 50 / 500 = 100 ms. Unbounded queues turn overload into outage.

One slow dependency

An order service runs at 500 QPS and calls a pricing service whose p99 is 50 ms.

L = 500 /s × 0.05 s = 25 concurrent

The pool holds 100, so utilisation is 25%. Four times headroom, every panel green.

Pricing now degrades to 2 s — a garbage-collection pause, a lost index, a hot shard. It is not down; down would be better. Arrival rate does not move — callers cannot tell anything changed. λ is still 500.

demand L = 500 /s × 2 s = 1,000 slots
supply       = 100 slots

• The pool is fully occupied after 100 / 500 = 0.2 s.
• Throughput collapses to 100 / 2 s = 50 QPS, so 450 requests per second pile up.
• Ten seconds in, the backlog is about 4,500 requests, most already older than the caller's own timeout.

The requests that die are not only the ones touching pricing. A worker is shared, so head-of-line blocking takes everything: the cache-only home endpoint needs a worker and there are none. The health endpoint queues too, so the load balancer pulls the instance and its 500 QPS lands on peers already in the same state. Upstream, the caller fills its own pool by the same arithmetic — one hop per pool, toward the edge.

slow dependency + no timeout
  ⟹ W unbounded
  ⟹ L = λW unbounded
  ⟹ pool exhausted
  ⟹ every endpoint sharing that pool fails
  ⟹ cascade upstream

How one slow dependency exhausts a worker pool and cascades to unrelated endpoints100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

The remedies are the law's three terms

Three variables, three levers — plus the option of splitting L into compartments.

Bound W with timeouts. The largest latency a pool of 100 survives at 500 QPS is W = 100 / 500 = 200 ms, which makes the timeout an arithmetic result rather than a preference: W_max = L_pool / λ_peak. At 200 ms it sits 4× above the healthy p99 of 50 ms, so it never fires in normal operation, yet the pool can never be over-subscribed. Propagate the remaining budget downstream, and budget retries too — three attempts at 200 ms is 600 ms of W and triple the λ, applied when the dependency is weakest.

Partition L with bulkheads. Give pricing 30 of the 100 workers. Healthy, it serves 30 / 0.05 s = 600 QPS, over the 500 needed. Sick at 2 s it holds at most 30, so 70 workers keep serving checkout and home. Peak utilisation drops in exchange for containment, the bargain made throughout fault tolerance.

Drive λ to zero with a circuit breaker. Timeouts still spend a worker for 200 ms per doomed call; at 500 QPS that is 100 workers burned on failure. A breaker that opens on a failure threshold sets λ to zero, so L = 0 × W = 0 and the compartment empties while a fallback — a stale cached price — answers. Mechanics and the half-open probe are in circuit breakers.

Cut λ at the door with load shedding. Admission capacity is λ_max = 100 / 0.2 s = 500 QPS. Above that, rejecting in about 1 ms costs a thousandth of admitting into a 30-second queue and keeps accepted traffic fast. Same control surface as rate limiting, pointed at the server rather than at client fairness.

In an interview

What is tested is whether a latency number and a resource number are connected in your head. "One of your dependencies gets slow — what happens?" is a standard probe: the weak answer is "that endpoint gets slower". The strong answer is "the whole service stops, including endpoints that never call it".

Say the numbers early: "at 500 QPS and 200 ms p99 we hold 100 requests in flight, so 100 is the floor for the pool; I would run 200 and cap the wait queue at 50." Then take it to failure unprompted: "if that dependency degrades to 2 s, demand is 500 × 2 = 1,000 against a pool of 100, so I need a 200 ms timeout, a 30-slot bulkhead, and a breaker."

Three mistakes lose points. Adding a retry with no timeout and no budget, which raises W and λ together. Treating a slow dependency as milder than a failed one — a refused connection returns in 1 ms and costs nothing, a slow one holds a worker. And sizing from mean latency, which understates L by whatever the tail is doing.

Check yourself

1. A service runs a 200-thread pool, p99 latency 400 ms, peak arrivals 600 QPS. Does the pool hold, and what is its ceiling?

No. Demand is 600 × 0.4 = 240 against 200 slots, so roughly 40 requests per second accumulate. The ceiling is 200 / 0.4 s = 500 QPS. Either cut W to 200 / 600 = 333 ms, or grow the pool to about 300 — but only if the dependency absorbs 300 concurrent calls, otherwise the queue just relocates.

2. Pool of 100, 500 QPS, dependency p99 50 ms. Choose a timeout and justify it in one line.

Worst-case occupancy is λ × timeout, so holding L at or under 100 requires 100 / 500 = 200 ms. That is 4× the healthy p99, silent in normal operation, and it caps occupancy even if the dependency stops answering entirely. Without it the dependency sets W, and at 2 s demand is 1,000.

3. Checkout calls a fraud service. Fraud p99 is 80 ms, checkout runs 300 QPS on a pool of 150 shared with every other endpoint. Fraud degrades to 3 s. How bad, and what changes first?

Demand becomes 300 × 3 = 900 against 150 slots; the pool is gone in 150 / 300 = 0.5 s and every unrelated endpoint on it dies too. The timeout goes first, the only remedy that works before anyone detects the incident: capped at 150 ms, worst-case fraud occupancy is 300 × 0.15 = 45 slots. Then a bulkhead pinning fraud near 50, then a breaker plus a decision on whether checkout fails open to manual review or closed.