Design a notification system

A password reset that never arrives is a support ticket. The same alert delivered five times is an uninstall. A notification system is judged almost entirely on those two failures, and both of them live in the retry path: third-party providers time out, rate-limit, and return ambiguous errors, so every send is attempted more than once. The design problem is making a repeat attempt harmless without making a lost attempt possible.

Step 1 — Requirements

Functional:

• Three channels — push (APNs for iOS, FCM for Android), SMS through a provider such as Twilio, email through SES. Each has a different rate ceiling and a different failure profile, which is why they cannot share a pipeline.
• Two traffic classes. Transactional is one recipient, triggered by an event, latency-sensitive: an OTP, an order confirmation, a password reset. Bulk is one template to millions of recipients, latency-tolerant but throughput-bound.
• Per-user preferences: opt-in per channel per category, quiet hours in the user's local timezone, and a global unsubscribe that must be honoured as a legal obligation rather than a feature.
• Templates rendered per locale, versioned.

Non-functional:

• No lost transactional notification. Password reset delivery is a durability requirement, not best effort.
• At most one perceived delivery per (user, notification, channel).
• Transactional p99 from event to provider handoff under 30 s; OTP under 5 s.
• A campaign must never delay transactional traffic. This single constraint shapes the architecture more than any other.
• One provider going down degrades one channel, not the system.

Out of scope: the in-app inbox, which is a read-path fan-out problem covered in design a news feed, and delivery/open analytics, which is a separate pipeline reading the same events.

Step 2 — Scale, shape, and what the average hides

Assume 10 million notifications/day.

10,000,000 ÷ 86,400 s ≈ 116/s, call it 120 QPS average. That number is close to useless on its own, because the two traffic classes have opposite shapes.

Assume the split is 6M transactional and 4M campaign on an ordinary day.

Transactional is 6,000,000 ÷ 86,400 ≈ 70 QPS average, with a normal diurnal peak of 3x — roughly 210 QPS. Flat, predictable, and small.

Campaign traffic is not a rate at all. A single send to the full 10M-user base is a day's entire volume injected in about a minute. If the fleet drains it in 400 seconds, the instantaneous rate is 10,000,000 ÷ 400 = 25,000 sends/s — around 350x the transactional baseline — sustained for under seven minutes and then gone. Sizing steady-state capacity for 120 QPS and then letting a campaign into the same pipe is how a password reset ends up 200 seconds late.

Fan-out arithmetic for that 10M send:

• Audience read in pages of 1,000 → 10,000 chunk tasks.
• Each chunk: one batched preference read of 1,000 rows, filter, render, then 2 provider multicast calls at 500 device tokens each. Roughly 2 s of wall clock, nearly all I/O.
• 10,000 chunks x 2 s = 20,000 worker-seconds. With 50 campaign workers: 20,000 ÷ 50 = 400 s ≈ 6.7 minutes.

The provider is not the bottleneck for push. At 25,000 recipients/s and 500 per call that is 25,000 ÷ 500 = 50 calls/s, and by Little's Law a 200 ms call needs 50 x 0.2 = 10 concurrent in-flight requests (Little's Law). Ten. Push is cheap because it batches.

SMS and email are the opposite, and the ceiling is contractual rather than technical. At a contracted 100 SMS/s, 4,000,000 ÷ 100 = 40,000 s ≈ 11 hours. At 200 emails/s, 4,000,000 ÷ 200 = 20,000 s ≈ 5.6 hours. Adding workers does nothing; it only collects 429s faster. The same campaign takes 7 minutes on push and 11 hours on SMS, which is the whole argument for a queue per channel rather than one queue with a channel field.

Two more sizings that decide storage:

Dedupe keys. One per (user, notification, channel), roughly 100 B including Redis entry overhead. 10,000,000 x 100 B = 1 GB per day retained; a 72-hour TTL gives 3 GB, comfortably one instance.

Queue payload. The message carries references, not output: user_id, notification_id, template_id, template_version, locale, channel, and a small parameter map — about 300 B. 10,000,000 x 300 B = 3 GB/day through the broker. Enqueuing rendered HTML email at 20 KB instead would be 10,000,000 x 20 KB = 200 GB/day, 66x the traffic, for output the worker could have produced itself. Render late.

Step 3 — The architecture

Notification pipeline from producers through prioritised queues to third-party providers100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

Why the queue is not negotiable

The naive version has the order service call SES inline and return when the email is accepted. It works in staging and fails in the first provider incident.

SES normally answers in about 300 ms at p99. During an incident it answers at the client timeout — say 10 s. The order service handles 500 QPS, so Little's Law gives 500 QPS x 10 s = 5,000 requests in flight against a thread pool of 200. The pool is exhausted in well under a second, and checkout is down because an email provider is slow (tail latency). A queue converts a provider's availability problem into our latency problem, which is the only trade in the whole design that is unambiguously worth making.

The second reason is retries. Inline, a send gets one attempt and the failure is lost with the request. Behind a queue the message is durable, survives a two-hour provider outage, and drains when the provider returns (message queues).

The outbox: the confirmation that was never queued

The bug is a dual write. The producer commits the order, then publishes the notification:

BEGIN; INSERT INTO orders ...; COMMIT;
publish("order.confirmed", ...)      # process dies here

The order exists and no email will ever be sent, and nothing in the system knows. Reversing the order is worse: publish first, transaction rolls back, and a customer receives a confirmation for an order that does not exist.

The fix is to write the notification intent into an outbox table inside the same transaction that writes the order, and let a relay poll that table and publish (the outbox pattern). One transaction, one commit, no window.

Size the relay: poll every 200 ms, batch up to 500 rows. At 70 transactional QPS each poll finds 70 x 0.2 = 14 rows — one query, nowhere near the batch limit. Added latency is bounded by the poll interval: 100 ms median, 200 ms worst, against a 30 s budget.

The relay is itself at-least-once. It can publish a batch and die before marking those rows dispatched, and on restart it publishes them again. That is not a defect to engineer away; it is the reason the next section exists.

Dedupe, and why at-least-once forces it

At-least-once delivery implies duplicates, and duplicates imply idempotency is required (idempotency). In this system duplicates enter from four places:

1. The outbox relay republishes an already-published batch.
2. The broker redelivers when a visibility timeout expires on a slow worker.
3. The worker crashes after the provider accepted the send but before it acked the queue.
4. The provider returns a timeout to us after having actually delivered.

Case 3 is the one candidates miss. The worker calls FCM, FCM returns 200, the worker is killed before acking. Thirty seconds later the message reappears. No broker feature prevents this — the HTTP call and the queue ack are in two different systems and cannot share a transaction. This is exactly why "the broker gives exactly-once, so I don't need dedupe" is wrong: exactly-once semantics end at the broker boundary, and the provider is outside it.

The key is hash(user_id, notification_id, channel). notification_id is stable per logical event — minted by the producer, written into the outbox row, carried through every republish — so all four duplicate paths present the same key.

The worker claims before sending:

# One atomic claim. NX means exactly one worker wins.
claimed = SET dedupe:<key> "sending" NX EX 120

if not claimed:
    ack_and_exit()          # someone else owns this send

send_to_provider(...)
SET dedupe:<key> "<provider_message_id>" EX 259200   # 72 h, terminal
ack()

The short EX 120 on the claim is deliberate. If a worker claims and then dies before sending, a permanent key would mean the notification is silently lost — the exact failure the design exists to prevent. A 120-second lease means a redelivery can take over an abandoned claim.

State the trade honestly: a lease of 120 s means a worker that hangs for 130 s and then completes produces a duplicate; a lease of an hour means a crashed worker loses that notification for an hour. Tune it per class rather than globally. Transactional wants the short lease and biases toward a duplicate — a second password reset email is noise. Bulk wants the long lease and biases toward a drop — a duplicated 10M-recipient push is a news story.

Preferences and quiet hours belong before the queue

At enqueue, the notification service resolves, in order: global unsubscribe, category opt-out, channel enablement, then quiet hours. The first three drop the message and record why. Quiet hours do not drop — they schedule. Convert 22:00–08:00 in the user's timezone into a send-after timestamp and put the message on a delayed queue.

Doing this in the worker instead is the mistake, and it costs three specific things. A 10M campaign where 40% have opted out enqueues 10M messages to send 6M, so queue depth stops being a measure of remaining work. Those 4M doomed messages each burn a worker cycle and a preference read: 4,000,000 x 2 ms = 8,000 worker-seconds, about 2.2 hours of compute to produce nothing. And a message that should never have existed still consumes its five retry attempts when it hits a transient error, so the retry budget and the dead letter queue fill with noise.

There is one honest caveat. Preferences change between enqueue and send, and on an 11-hour SMS campaign a user can unsubscribe mid-drain — an unsubscribe that must legally be honoured. So the worker does one cheap final check against a Redis set of opt-outs: a single lookup, no join, no join to the preferences database. That is a re-check, not the primary filter.

Quiet hours apply to marketing only. An OTP at 03:00 was requested at 03:00.

Provider rate limits, backoff, and the circuit breaker

Each provider gets its own token bucket held in Redis and shared across workers, so the fleet has one honest count rather than 50 local ones (design a distributed rate limiter). Set capacity to about one second of rate — enough to absorb a jitter burst, not enough to sustain an overrun. Concretely: SES at 200/s, Twilio at 100/s, FCM at 600,000 messages/minute which is 10,000/s.

A worker that cannot get a token does not spin. It returns the message to the queue with a visibility delay, so the queue stays the buffer and worker memory does not become one.

Retries use exponential backoff with full jitter: nominal delays of 1, 2, 4, 8, 16 s across five attempts, 31 s of retrying in total, with each actual delay drawn uniformly between zero and the nominal. Without the jitter, several thousand workers that failed at the same instant retry at the same instant and reproduce the overload they were backing off from.

Classify errors before retrying anything. A 429 or a 5xx is retryable. A 400 malformed payload, a 403, and a 404 unregistered device token are not — retrying them is pure waste. Unregistered tokens get their own path that deletes the token, because retrying a dead token five times a day forever is how a device-token table rots.

Above that sits a per-provider circuit breaker: if APNs errors exceed 50% over a 30-second window, open for 30 s, then half-open with a trickle of probes. While the circuit is open, messages stay queued rather than spending their retry budget on a provider that is known to be down.

The dead letter queue

After five attempts a message moves to a dead letter queue instead of cycling forever. Without one, a single poison message — a template that throws on a null field, a payload the provider rejects as too large — is redelivered indefinitely and permanently occupies a worker slot.

Alert on rate, not depth. At 10M/day, a 0.1% failure rate is 10,000,000 x 0.001 = 10,000 messages/day landing in the DLQ, which is a normal background of expired tokens and bad addresses. A transactional DLQ, by contrast, should be empty; page on any depth at all there. Once the cause is fixed, redrive the DLQ back into its source queue — where the dedupe keys make the replay safe.

Templates and localisation

The worker renders. The queue message carries template_id plus template_version plus parameters — the 300 B against 20 KB from Step 2.

Pin the version at enqueue. A campaign that takes seven minutes to drain must not change its wording halfway because someone saved an edit, and a job that takes eleven hours definitely must not.

Locales resolve down a fallback chain — pt-BR then pt then en — so a missing translation renders the fallback rather than a raw key. For SMS, measure after rendering: a segment is 160 GSM-7 characters, and a translated string that spills to two segments doubles the bill; one non-GSM character drops the segment to 70 UCS-2 characters and can triple it. Check at template-publish time, not at send time.

Render failures are not retryable. The same template with the same parameters throws the same exception five times. Straight to the DLQ.

Fan-out for bulk, and the classic mistake

The classic mistake is one queue for everything, and it is worth pricing exactly. A 10M campaign lands, the fleet drains the shared queue at 25,000/s, and a password reset enqueued behind it sits behind an average of 5M messages: 5,000,000 ÷ 25,000 = 200 s typical, 400 s worst. The budget was 30 s. FIFO does not know that one message is a marketing push and the other is a login code.

Three layers fix it.

Separate queues by class and channel. Six of them: push.transactional, push.bulk, sms.transactional, sms.bulk, email.transactional, email.bulk. Six independent backlogs, and the 11-hour SMS campaign backlog is invisible to the OTP path.

Separate worker pools with reserved capacity. Transactional workers never read a bulk queue. A pool that can be borrowed will be borrowed at exactly the wrong moment, so remove the option rather than tuning a priority weight.

Split the provider budget by class. Of the 200 emails/s SES allows, reserve 50/s for transactional and configure the bulk token bucket at 150/s. A campaign then cannot consume transactional headroom, because it physically cannot obtain the tokens.

Chunking makes the campaign restartable as well as throttled. The expander pages the audience query 1,000 rows at a time and emits 10,000 chunk tasks, each carrying its cursor; chunk workers expand a task into per-user sends. The broker sees 10,000 messages instead of 10M, so enqueue finishes in seconds, and a crash re-runs one chunk rather than the campaign. Capping bulk chunk concurrency at 50 workers bounds the campaign's instantaneous rate by construction rather than by hope.

Step 4 — Failure modes and trade-offs

A provider is down. The circuit opens and messages accumulate. Transactional push at roughly 40 QPS through a 30-minute outage is 40 x 1,800 = 72,000 messages, which at 300 B is 22 MB — the backlog is trivial to hold. The real question is the drain, and the token bucket already answers it. Give time-sensitive notifications a TTL: a "your ride is arriving" push is worthless 30 minutes later, so drop it rather than deliver it stale. Password resets keep, and expire on their own terms.

The dedupe store is unavailable. Fail open for transactional, fail closed for bulk. It is a per-class policy, not a global one, for the same reason the lease length is.

Someone clicks Send twice on a campaign. Per-message dedupe does not save you, because the second launch mints fresh notification_ids. The campaign API needs its own idempotency key on the create call.

Decision	Cheap option	Correct option	Take the cheap one when
Send path	Call the provider inline from the producer	Enqueue, worker calls the provider	Never — one 10 s provider timeout at 500 QPS puts 5,000 requests in a 200-thread pool
Producer to queue	Publish after commit	Outbox row in the same transaction, relay publishes	The notification is genuinely best effort, like a "new follower" badge — never for a receipt
Duplicate control	Trust the broker's exactly-once	Dedupe key per (user, notification, channel), claimed with SET NX	Never — exactly-once ends at the broker, and the provider call is outside it
Queue layout	One queue, channel as a field	Six queues: channel x class, separate pools	Bulk campaigns do not exist in the product and never will
Preference check	Filter in the worker	Filter at enqueue, cheap re-check in the worker	Opt-out rates are low single digits and there is no bulk traffic
Payload	Enqueue rendered content	Enqueue a pinned template reference, render in the worker	Payloads stay under ~1 KB and templates never change mid-drain
Terminal failure	Retry until it succeeds	Five attempts with jittered backoff, then DLQ	Never — one poison message occupies a worker permanently

In an interview

What is being tested is whether you can reason about a system whose correctness depends on components you do not control. Providers fail, rate-limit, and occasionally lie about whether they delivered. Every candidate draws producers to a queue to workers inside two minutes; that drawing is table stakes, and the entire signal is in what you say about retries.

Lead with the shape rather than the average. "Ten million a day is 120 QPS average, but the average is meaningless here. Transactional is flat at about 70 QPS with a 3x peak. A campaign is 10M messages in seven minutes, roughly 350x the baseline. So the first structural decision is a queue and a worker pool per channel per class, because otherwise FIFO puts a password reset behind five million marketing pushes and adds 200 seconds to it."

Then own the reliability chain out loud, because nobody will hand it to you: at-least-once delivery implies duplicates implies a dedupe key per (user, notification, channel), claimed with SET NX and a short lease so a crashed worker cannot lose the send; plus an outbox, so a committed order can never end up without its confirmation.

Three mistakes lose points reliably. The first is putting the preference check in the worker — interviewers ask "where do you check opt-out?" precisely because the wrong answer shows you are reasoning about one message rather than ten million. The second is claiming the broker's exactly-once removes the need for dedupe; name the crash between the provider's 200 and the queue ack and the claim collapses. The third is retrying a 429 immediately, or retrying it without jitter, which synchronises the fleet and reproduces the overload.

Volunteer the dead letter queue and the reserved transactional throughput before you are asked. Both are cheap to say and both signal that you have operated one of these.

Check yourself

1. Everything shares one queue. A 10M-recipient campaign is launched. The transactional p99 budget is 30 s. Compute the damage and decide what to change.

The campaign injects 10M messages and the fleet drains about 25,000/s, so a full drain is 10,000,000 ÷ 25,000 = 400 s. A password reset enqueued just after the campaign waits behind roughly 5M messages: 5,000,000 ÷ 25,000 = 200 s typical and 400 s worst — 6.7x to 13x over budget. Split into per-channel, per-class queues with separate worker pools, and reserve a slice of the provider rate for transactional. The campaign's own drain time does not change; it simply stops being in front of the reset.

2. A 4M-recipient SMS campaign must go out in a four-hour evening window. The provider contract allows 100 messages/s. Does it fit, and what do you do?

4,000,000 ÷ 100 = 40,000 s ≈ 11.1 hours. It does not fit, and adding workers cannot help — the ceiling is the contract, not our concurrency, so extra workers only collect 429s. Real options: raise the contracted rate to 4,000,000 ÷ 14,400 s ≈ 280/s, segment the audience across three nights, or move the campaign to push, where the same 4M drains in a couple of minutes. Whichever you pick, it runs on sms.bulk, because eleven hours of backlog on a shared SMS queue makes every OTP that night late.

3. The dedupe store loses all its keys mid-morning. Keep sending or stop?

Answer per class. Transactional: keep sending. The window that matters is seconds, most in-flight messages have no duplicate pending, and the failure dedupe prevents — an annoying second email — is much cheaper than the one stopping causes, which is a missing password reset. Bulk: stop. A campaign that resumes after losing its keys re-sends to everyone already delivered, and at 10M recipients that is an incident. Reduce the exposure structurally: persist the terminal dedupe record alongside the send record for transactional, and checkpoint chunk completion per campaign so a restart resumes at chunk 6,412 rather than chunk 1.