Fault Tolerance
Budget the seconds a failover actually costs, size bulkheads from a pool calculation, and say what users see between the failure and the recovery.
Redundancy gives you a spare. Fault tolerance decides whether that spare actually takes over — automatically, fast enough to matter, and without corrupting state on the way. The two are routinely confused, and a design with replicas but no tested failover has bought neither.
Redundancy is the technique, fault tolerance is the property
Fault tolerance is the property that lets a system keep operating properly — possibly at a reduced level, which is graceful degradation — when one or more components fail. A component failure must not become a system failure.
Redundancy is one technique for getting there: duplicate instances, replicated data, a second network path, a second availability zone. Fault tolerance is that redundancy joined to detection, switchover and recovery. A cluster with a follower nobody promotes is redundant and not fault tolerant, and the gap costs money twice — once for the spare, once for the machinery that uses it.
Detection: the timeout you pick is an availability decision
Three detectors do most of the work: health checks run by a load balancer against a service endpoint, heartbeats sent to a monitor, and metrics — error rate, latency, saturation — which catch the failures that answer a health check perfectly while serving garbage.
Every detector trades a false negative against a false positive, and the timeout is the knob. In the diagram the monitor heartbeats every 2 s and declares death after three missed beats: roughly 6 s to notice. Tighten that to one missed beat and detection drops to 2 s — but a 3-second stop-the-world garbage collection pause on a healthy leader now looks exactly like death. Promoting a follower while the old leader still takes writes is worse than the fault itself: two nodes believe they lead, both accept writes, and the divergence is reconciled by hand. That is split-brain, and the defence is a quorum to promote plus a fencing token the deposed leader cannot forge — what consensus protocols are for.
Detection also has to sit outside the failure domain: a check served by the same exhausted thread pool as real traffic tells you nothing. So checks come from the load balancer, heartbeats go to a separate monitor, and metrics live in monitoring that outlives what it watches.
Failover, and the seconds it costs
Three switchovers, in ascending order of cost:
- Load balancer failover — a failing instance leaves the pool after N failed checks. Seconds, and invisible when requests are retried.
- Database failover — a follower is promoted to leader and the app tier learns the new address.
- DNS failover — records repointed at a healthy address or a second datacenter.
"We fail over automatically" is not a number. The diagram's sequence costs 6 s to detect, 5 s to promote and 2 s for app servers to pick up the new address: about 13 s of failed writes. A 99.99% target allows 0.01% of 30 days — 4.3 minutes, about 259 s, a month — so one event spends 5% of the budget and roughly nineteen fit inside it.
Route the same failover through DNS with a 300 s TTL and the last term swallows the other two: 6 + 5 + up to 300 ≈ 311 s, 5.2 minutes — one failure spending the whole four-nines month. DNS failover is therefore a region-level tool; below that, put a load balancer in front and let health checks act in seconds. The tempting alternative is worth rejecting out loud: a 5 s TTL buys resolver traffic all year to shorten one event, and many resolvers ignore short TTLs anyway.
Promotion also costs data. A leader that acknowledges writes locally and ships them asynchronously leaves the follower 200 ms behind, so at 300 writes/s promotion drops 300 × 0.2 = 60 acknowledged writes. Synchronous replication removes that loss and buys it back as a round trip on every commit, so the question is what share of the write path's latency budget that round trip takes. Assume checkout is held to a 300 ms p99 end to end. A follower in the same availability zone costs one 0.5 ms intra-datacenter round trip — 0.2% of that budget, which no user can feel. A follower in another region costs a ~200 ms India-to-US-East round trip on every write — two thirds of the budget spent on replication before the request does any work of its own. Sync near, async far — a consistency decision wearing a durability hat.
Sync near leaves a residual worth naming: a synchronous follower in the same zone survives the loss of a node, not the loss of the zone. If the availability zone goes, the acknowledged writes only that zone held go with it. A ledger that has to survive that puts the synchronous follower in a second zone of the same region, where the round trip is still the low-millisecond order of the 0.5 ms figure rather than the ~200 ms a second region costs.
Isolation: bulkheads stop one failure becoming all of them
A ship's bulkheads let one compartment flood without sinking the vessel. In a service the compartments are resource pools.
An app tier holds a 200-connection pool and calls two dependencies: checkout and recommendations. Recommendations degrades from 50 ms to 2 s while still taking 100 requests/s. Little's Law — concurrency = arrival rate × latency — gives 100 × 2 = 200 connections in flight. The degraded, non-critical dependency owns the entire pool, and checkout, which never failed at all, times out.
Give recommendations its own pool and the same event is capped. Healthy it needed 100 × 0.05 = 5 connections, so a pool of 20 leaves 4x headroom and holds the blast radius to 10% of the tier; checkout keeps 180. Add a circuit breaker and even those 20 stop waiting 2 s to fail. A microservice boundary gives the same isolation by construction; one pool per dependency inside a monolith gets most of it for far less.
Degrade rather than fail
Graceful degradation keeps the core functions working while non-critical parts are unavailable: an e-commerce site whose recommendation service dies drops personalised recommendations and keeps search and checkout serving. The work is ranking features in advance — which may serve stale data from cache, which may return an empty result, and which must fail outright. A payment authorization is never degraded into a success.
Retry safely, or not at all
Stateless services make failure cheap: no context died with the machine, so any healthy instance can serve the retry. Retries with exponential backoff absorb transient faults, and the backoff needs jitter, or every client retries in the same instant and rebuilds the spike.
Retries are also how a system becomes at-least-once: a request that timed out may already have been processed, and the retry duplicates it. At-least-once means duplicates, and duplicates mean idempotency is required — a key on every unsafe endpoint, so the second attempt at a payment returns the first result instead of charging twice.
In an interview
What is being tested is whether you know how your design fails, in seconds, and what the user sees during that window. A second box is not the signal; the failure, the detector, the switchover time and the user-visible effect are.
Point at one single point of failure in your own diagram — usually the leader database, the load balancer, or a single availability zone — then say something in this shape: "The load balancer health-checks every 2 s and ejects after three failures, so a dead app server drains in about 6 s and client retries cover it. For the database the monitor promotes the follower — call it 10 to 15 s of failed writes, and the last 200 ms of unreplicated writes are lost, which is why payments replicate synchronously to a follower in the same availability zone; that follower survives a dead node, not a dead zone. During that window reads come from the follower and writes return 503 with Retry-After. Those reads are up to 200 ms stale, so read-your-writes breaks: a user who changed their delivery address a moment before the failure reloads and sees the old one. So a session that has just written is pinned to the leader for a few seconds afterwards instead of being served by any replica." Then price it against the target: 99.9% is 43 minutes a month, 99.99% is 4.3, and failover time × expected events is what you spend.
The mistake that loses points is naming redundancy and stopping there — "we have a replica, so the database is highly available." It answers none of the questions being asked: who notices, how long detection takes, what happens to in-flight writes, how clients learn the new address. Its close relative is claiming failover is instant. If your number is zero seconds you have not thought about detection, and the next question will be about a GC pause.
Check yourself
1. Your target is 99.99%. Detection takes 6 s, promotion 5 s, and clients find the database through DNS with a 300 s TTL. Do you meet the target, and what do you change?
Not with that TTL. One event costs 6 + 5 + up to 300 ≈ 311 s ≈ 5.2 minutes, while 99.99% of a 30-day month allows 0.01% × 43,200 minutes = 4.32 minutes, about 259 s in total — one failover overspends the month. Move client discovery off DNS to a load balancer repointed in a second or two: the event costs about 13 s, leaving room for roughly nineteen a month. A 5 s TTL is the wrong fix — year-round resolver load to shorten one event, and short TTLs are widely ignored.
2. One app tier, a 200-connection pool, two dependencies. Recommendations runs at 100 requests/s and degrades from 50 ms to 2 s. What breaks, and how big is the bulkhead you give it?
concurrency = 100 × 2 = 200, so the degraded dependency takes the whole pool and checkout times out despite never failing. Healthy recommendations needed 100 × 0.05 = 5 connections, so a pool of 20 gives 4x headroom and holds the damage to 10% of the tier. Checkout keeps 180, and a non-critical failure stays non-critical. Pair it with a breaker so those 20 fail fast instead of sitting on a 2 s timeout.
3. The write path runs at 300 writes/s and the follower is 200 ms behind. Should replication be synchronous?
Promoting that follower loses 300 × 0.2 = 60 acknowledged writes. For a feed of likes, accept it and stay asynchronous. For a payment ledger, do not: those writes go synchronously to a follower in the same availability zone, a 0.5 ms round trip per commit, 0.2% of a 300 ms p99 write path. What you do not do is replicate synchronously across regions, where a ~200 ms India-to-US-East round trip takes two thirds of that budget on every write. Say the residual out loud too: same-zone sync survives a dead node, not a dead zone, so a ledger that must survive an AZ loss puts the synchronous follower in a second zone of the same region. Sync near, async far — and say which writes are which.