Circuit breakers: why a slow dependency is worse than a dead one
Size the timeout, trip threshold and open window of a breaker from real latency numbers, and pick the fallback that serves traffic while it is open.
A failed dependency is survivable — the call returns an error and you move on. A slow dependency is not: every request waiting on it holds a connection, and by Little's Law a jump from 50 ms to 2 s multiplies the connections in flight fortyfold. The pool drains, healthy traffic queues behind sick calls, and a service that never failed goes down anyway. A circuit breaker is the switch that stops calling a dependency once it is clearly unwell.
The failure it prevents
Service A takes 500 requests per second and calls Service B once per request. B normally answers in 50 ms, so 500 × 0.05 = 25 calls sit in flight at any instant, and a 200-connection pool looks generous.
Then B degrades. It does not crash — it slows until calls hit the 2 s client timeout. Arrival rate has not changed, so the same arithmetic gives 500 × 2 = 1,000 calls in flight against 200 connections. The pool is empty after the first 200 requests, which at 500 QPS takes 0.4 s. Everything behind them queues, including calls to perfectly healthy dependencies that share the pool. A's own p99 blows past its budget, its health checks time out, the load balancer pulls instances that were never broken, and A's callers repeat the story one level up. That is a cascading failure, and nothing in it crashed.
Retries make it worse at the exact moment B can least afford it. Two retries per call triple the offered load: a dependency already failing at 500 QPS is now asked for 1,500. Retries need a budget — a cap on the share of traffic allowed to be a retry — plus exponential backoff with jitter, and because at-least-once delivery produces duplicates, idempotent handlers on the far side.
The three states, and what each one buys
Closed is normal operation: calls pass through and the breaker classifies outcomes. Timeouts, connection errors and 5xx responses count as failures; a 400 or a 404 does not, because a client error says nothing about the dependency's health — a breaker that counts 4xx trips itself the first time a user sends bad input.
Open rejects every call without touching the network, and this is the state doing the work. The 2 s timeout becomes a sub-millisecond error, so in-flight concurrency falls from 1,000 back to near zero and the pool is free for everything else the service does. The dependency also has its load removed while it recovers, which matters when the cause was overload rather than a bug.
Half-open, once the open window expires, lets one or two probes through. Success closes the breaker and resets the counters; one failure re-opens it and restarts the timer. The limit is the point: if all 500 QPS resumed the instant the timer fired, the stampede would knock the recovering dependency straight back over.
Picking the numbers
Timeout first. A breaker counts failures, and a call with no timeout never produces one — it produces a thread that waits forever. Derive it from the dependency's healthy p99, roughly 3×: p99 of 80 ms gives a 250 ms timeout. Set it too tight and normal tail requests become failures that trip the breaker on a healthy dependency; too loose and it stops bounding concurrency, which was the whole point.
Trip on a rate over a volume floor, not on a bare count. A fixed count means a different error rate at every traffic level, so it cannot survive a change in volume. The diagram's five failures in 10 s is a sensible trigger at 12 QPS, where it means about 4% of the 120 calls in that window. Take the same rule to 500 QPS and it fires on five failures out of 5,000 — a 0.1% error rate, which is ordinary background noise, so the breaker opens on a dependency that is essentially healthy. Go the other way, below half a call per second, and the window holds fewer than five calls, so the same count is unreachable even at a 100% failure rate. A bare percentage over-trips from the opposite end: one failure out of one request at 3am is 100%, and a single unlucky call is not a signal. Use both: 50% failures over a rolling 10 s window, evaluated only once at least 20 calls have landed in it. The rate scales with traffic; the floor discards the windows too small to mean anything.
That floor has a failure mode worth knowing, because breaker state usually lives per process. With 40 instances of A each seeing 500 ÷ 40 ≈ 12 QPS, a 10 s window holds ~120 calls and the floor is met easily. A low-traffic admin endpoint at 6 QPS total is a different story: 0.15 QPS per instance is 1.5 calls per window against a floor of 20, so its breaker never trips even at a 100% failure rate. Move the counter somewhere shared, or accept that a timeout plus a bounded retry is the whole design.
Open duration is a bet on recovery time. 30 s is a reasonable default: long enough for a process restart or a garbage-collection pause to finish, short enough that recovery is noticed quickly. Costs are symmetric — at 500 QPS a 30 s open window fails 15,000 requests fast, while a 1 s window sends a probe wave every second and keeps a struggling dependency pinned.
What "open" actually returns
Failing fast is only half the design; the other half is what the caller gets instead. A stale cached value is usually best where the data tolerates it — a price from 40 s ago beats no page. A degraded default works when the feature is additive: an empty recommendations strip, an unpersonalised feed. A write can be queued and applied later, provided the client is told it was accepted rather than completed. Otherwise return 503 with Retry-After, so callers do not retry into the open window.
The rejected alternative is worth saying out loud: no fallback, an error straight to the user. It still beats the pool exhaustion it replaces, but it turns one sick dependency into a visible outage of your own service, so it belongs only where staleness is genuinely unacceptable — a payment authorisation, not a product page.
Where it lives, and what it is not
Client libraries (Resilience4j, Polly, formerly Hystrix) put the breaker in the calling process, where the pool it protects lives. A service mesh sidecar moves it out of your code and centralises the configuration, but it does not change the arithmetic: a sidecar runs one per caller pod, so Envoy-style outlier detection and circuit breaking are enforced per proxy instance, and each one still sees only the traffic its own process sends. What actually aggregates counts across instances is a hop every caller genuinely shares — an API gateway, or a central proxy in front of the dependency — or counter state held somewhere shared, such as Redis. That is the fix for the low-traffic floor above, paid for with an extra hop and a component that knows nothing about the caller's own state.
A breaker is not a rate limiter. Both shed load, in opposite directions: a rate limiter protects a service from callers asking for too much, a breaker protects a caller from a dependency that has stopped answering. Nor does it replace a bulkhead — separate pools per dependency, so one saturated pool cannot starve calls to anything else. Timeout, bulkhead, breaker, retry budget: four controls, and the breaker works only because the other three are there.
In an interview
What is being tested is whether you understand that slow is more dangerous than dead, and whether you can attach numbers to a pattern instead of reciting three state names. Anyone can list closed, open and half-open; the signal is in the parameters and what each one costs.
Say it concretely: "I'd wrap the pricing-service call in a breaker — 250 ms timeout, roughly 3× its 80 ms p99 — tripping at 50% failures over a rolling 10 s window with a 20-call minimum, open for 30 s, then a single probe. While open we serve the last cached price and label it stale." That answer names a trigger, a duration, and a fallback, which is three more decisions than most candidates make.
The mistake that loses points is describing the states without ever naming a timeout. A breaker on a call that can hang forever never trips, because a hung call is not a failure — it is an unbounded wait, and the pool fills anyway. The second-most common is putting a breaker on the primary database, where "open" means every request fails and there is no fallback to serve; there the answer is a bulkhead, load shedding and a read replica, not a breaker.
Check yourself
1. Your service takes 400 QPS against a dependency with a 60 ms p99 and a 200-connection pool. The dependency degrades to a flat 3 s. How fast does the pool fill, and what timeout would have contained it?
Little's Law:
400 × 3 = 1,200calls in flight against 200 connections, so the pool is six times oversubscribed. It fills as soon as 200 requests have arrived —200 ÷ 400 = 0.5 s. A 200 ms timeout (roughly 3× the 60 ms p99) caps in-flight calls at400 × 0.2 = 80, comfortably under 200, so the pool survives even before the breaker trips. Set the timeout first; the breaker then stops the 80 wasted slots as well.
2. The breaker is per process across 60 instances, the endpoint takes 6 QPS in total, and it trips at 50% failures over a 10 s window with a 20-call floor. Does it ever open?
Each instance sees
6 ÷ 60 = 0.1QPS, so 1 call per 10 s window against a floor of 20. It never opens, even at 100% failure. Three options: widen the window (20 calls at 0.1 QPS needs 200 s, by which point the signal is stale), lower the floor (and accept tripping on noise), or move the counter somewhere all 6 QPS is one stream — a gateway hop every caller shares, or counters in Redis. A per-pod mesh sidecar is not that: it sees the same 0.1 QPS the in-process library did. At this volume the honest answer is often a tight timeout and a single retry, and no breaker at all.
3. Open for 30 s at 500 QPS. How many requests does that window fail, and what does the number tell you to decide first?
500 × 30 = 15,000requests fail fast per trip. That number is only tolerable if the fallback is, so decide the fallback before the duration: 15,000 stale-but-served responses is a non-event, 15,000 hard 503s is an incident. If no fallback exists, shorten the open window toward 5 s and accept six probe waves in the same 30 s — and compare against the alternative, which is those same 15,000 requests each holding a connection for the full timeout and taking the rest of the service down with them.