Design a news feed

A news feed merges many small write streams into one ordered read stream, and every interesting decision in it falls out of that shape. Writes are cheap and numerous, reads are frequent and sit on the user-facing latency path, and the mapping between them is one-to-many with a fan-out factor that varies by five orders of magnitude between a normal account and a celebrity. Get the fan-out strategy right and the rest is plumbing. Get it wrong and a single post stalls the cluster.

Step 1 — Requirements

In scope:

• A user sees a feed of posts from the accounts they follow, newest first. Ranking arrives later, as a layer on top.
• A user can post: short text plus an optional media reference.
• A user can follow and unfollow another user.

Out of scope: ads, stories, direct messages, notifications, search.

Non-functional targets, as numbers rather than adjectives — the habit from gathering requirements:

• Feed load p99 under 200 ms. The mean hides the tail; p99 is what users feel.
• A post reaches its followers within a few seconds. Other people's posts may be stale; your own may not.
• Availability over consistency. A stale feed is fine, a blank feed is not. When a partition occurs we keep serving possibly-stale feeds; PACELC's Else branch covers the rest of the time, where we still choose latency over consistency and read the nearest replica. See consistency models.

Step 2 — Estimates

Assume 500M daily active users, 2 posts per user per day, 10 feed loads per user per day, and an average of 200 followers and 200 followees.

Post writes. 500M × 2 = 1B posts/day. One billion requests a day is about 12,000 QPS average, so writes land at ~12,000 QPS, ~36,000 at 3× peak.

Feed reads. 500M × 10 = 5B feed loads/day = 5 × 12,000 ≈ 60,000 QPS average, ~180,000 QPS at peak. The read-to-write ratio at the request level is only 5:1, which is unremarkable. The interesting ratio is elsewhere.

The fan-out multiplier. Every post must reach 200 feeds:

1B posts/day × 200 followers = 200B feed writes/day
200B/day = 200 × 12,000 QPS ≈ 2.4M feed writes/second average
peak ≈ 7M/second

That number decides the architecture. A commodity Postgres box handles roughly 5,000 simple QPS, so 2.4M ÷ 5,000 = 480 boxes doing nothing but appending IDs, before replication. In Redis an append is an in-memory operation — 100 ns of actual work, dominated by the 0.5 ms datacenter round trip and amortised by pipelining — and one node absorbs on the order of 100,000 ops/s: 2.4M ÷ 100,000 = 24 nodes, call it 72 provisioned for peak. Twenty-four versus 480 is the entire reason the feed store is memory-resident and not a table.

Storage of precomputed feeds. Cap each feed at the 500 most recent post IDs. A post ID is 8 bytes; a sorted-set entry also carries a score and per-entry overhead, so budget ~20 bytes per entry.

500 entries × 20 bytes   = 10 KB per user
10 KB × 500M users       = 5 TB
5 TB ÷ 100 GB per node   ≈ 50 nodes

Why cap at 500: at 20 posts per screen that is 25 pages, further than almost anyone scrolls; past the cap, fall back to a pull query. Uncapped, a feed grows by 200 followees × 2 posts = 400 IDs/day, ~146,000/year, ~2.9 MB per user, ~1.5 PB in total. The cap converts unbounded growth into a fixed 5 TB.

Why IDs and not content. A hydrated post is ~1 KB: text, author, timestamps, counters, media refs. Store content in each feed and you store it 200 times: 500 × 1 KB × 500M = 250 TB, a 50× increase for the same data. Worse, an edit or delete must find and rewrite 200 copies. IDs are immutable, content is not — denormalise the immutable thing and hydrate the mutable thing at read time from one authoritative row.

Step 3 — The central decision: push or pull

Fan-out on write (push). At post time, look up the author's followers and append the post ID to each of their feeds. The write costs 200 operations; the read is a single range read, ~0.5 ms. The cost is paid at write time whether or not anybody reads.

Fan-out on read (pull). At post time, append once to the author's own timeline. At read time, fetch each followee's timeline, merge, sort, truncate. The write costs 1 operation; the read costs 200 queries — 60,000 feed QPS × 200 = 12M timeline queries/second, all on the latency path. Even fully parallelised, the p99 of a feed load becomes the p99 of the slowest of 200 calls, far worse than the p99 of one.

For the average account push wins outright: 200 async writes against one sync read.

The celebrity problem. An account with 50M followers posts once: 50M feed writes. Against a cluster provisioned for 2.4M writes/second, one post consumes

50,000,000 ÷ 2,400,000 ≈ 21 seconds

of the entire cluster's write capacity. And most of that work is thrown away: if 10% of those followers open the app today, 45M of the 50M writes land in feeds nobody reads.

Pull is at its best in exactly this case. All 50M followers read the same author timeline: one key, extremely hot, cache hit ratio near 100%, one list served out of memory. The property that makes push catastrophic — a huge follower count — is the property that makes the pulled key perfectly cacheable (caching covers why hot keys are the easy case).

Push and pull fail at opposite ends of the same distribution. That inversion is the answer.

The hybrid

Fan out on write when the author has fewer than a threshold number of followers. Above it, skip fan-out and let readers pull from the author timeline, merging the two sources at read time.

Hybrid fan-out: precomputed feeds for normal accounts, read-time pulls for celebrities100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

Ten thousand followers is a defensible starting threshold. It bounds worst-case fan-out at 10,000 writes per post — about 0.1 s of one Redis node, off the latency path because it runs on a queue — and above it the author timeline is hot enough for a cache to absorb the reads. Treat it as a tunable.

Read path for a user following 200 accounts, about 5 of them above the threshold:

1. Range read the precomputed feed by cursor: 20 IDs, 0.5 ms.
2. Fetch recent IDs from each of the 5 celebrity timelines in parallel: 0.5 ms, served from cache.
3. Merge by score, truncate to 20.
4. Hydrate 20 post IDs in one multi-get against the post cache: 0.5 ms, misses fall through to the post store.

Four sequential round trips is ~2 ms of network, leaving the rest of a 200 ms p99 budget for TLS, auth, ranking and serialisation.

The fan-out queue is at-least-once, so duplicates are guaranteed and fan-out workers must be idempotent. ZADD feed:<user> <ts> <post_id> is naturally idempotent — reapplying it sets the same member to the same score. LPUSH is not: replay it and the post appears twice. The message queues lesson works the general chain — at-least-once implies duplicates implies idempotency required.

Deletes and unfollows are read-time filters, not write-time corrections. A deleted post hydrates to a tombstone and is dropped; an unfollowed author's entries are filtered on read and age out past the 500 cap. Same trade both times: the write-time fix costs 200× the read-time one.

One consistency exception matters: your own post must appear in your own feed immediately. Write to the author's feed synchronously before returning, and let async fan-out handle everyone else. If feed reads come from a follower replica, replication lag breaks read-your-writes and the user watches their own post vanish on refresh — pin the author's reads to the leader for a short window after each write.

Step 4 — Deep dives

The feed store

One Redis sorted set per user: key feed:<user_id>, member = post ID, score = post timestamp (or a snowflake ID that already sorts by time). A sorted set rather than a list because:

• A range read by score is cursor pagination, in one command.
• ZADD is idempotent under queue replay; LPUSH is not.
• Trimming to the cap is a single ranged removal, O(log N + trimmed).

The feed store is derived data, not a source of truth. Losing a shard means rebuilding those feeds from followee timelines — slow, but correct and automatic. That is why it can live in memory with modest persistence and no cross-region replication, while the post store cannot.

Shard the feed store by user ID: feed reads are single-key lookups, so hash partitioning spreads load evenly with no cross-shard reads on the hot path — the clean case in database scaling. The follow graph needs two indexes over the same 100B edges (500M × 200 follows ≈ 1.6 TB at 16 bytes each): followee-to-followers for push, follower-to-followees for pull.

Pagination: cursor, not offset

Offset pagination breaks because the feed changes underneath the reader. Page 1 returns items 0–19; while the user reads, 3 new posts arrive at the head and shift everything down. OFFSET 20 LIMIT 20 now returns what were items 17–36, so items 17, 18 and 19 are shown a second time. Deletions produce the mirror bug: items shift up and are skipped, never seen at all. The symptom is "my feed repeated itself", and it never reproduces on a quiet test account.

A cursor names a position in the ordering rather than a count from the head: the sort key of the last item returned, the pair (timestamp, post_id), with the ID as tiebreak because at 12,000 posts/second timestamps collide constantly. The next page is "the 20 items whose key is strictly less than the cursor". Inserts at the head do not move that boundary, so no duplicates and no gaps.

Cost points the same way. OFFSET 10000 makes the store walk and discard 10,000 entries, so page 50 costs 50× page 1, while a cursor is an O(log N) seek at every page. Encode it opaquely — base64 of the tuple — so the sort key can change later without breaking deployed clients.

Ranking as a later layer

Do not rank inside fan-out. Keep retrieval and ranking separate:

• Retrieval is the hybrid above: the ~500 most recent pushed IDs plus the pulled celebrity posts, roughly 600 candidates.
• Ranking scores those candidates at read time on author affinity, recency and predicted engagement, and returns the top 20.

Three reasons. Ranking models change weekly; the feed store must not. A score computed at write time is stale before it is read, because engagement counters keep moving. And the arithmetic forces a two-pass ranker: 60,000 QPS × 600 candidates = 36M scorings/second, affordable only for a cheap first pass — a linear model or small tree ensemble — with a heavier model on the surviving top 50. Cache the ranked page for ~30 s so a pull-to-refresh does not re-rank.

Trade-offs

Dimension	Push (fan-out on write)	Pull (fan-out on read)	Hybrid
Write cost per post	200 feed writes; 50M for a celebrity	1 append	200 below threshold, 1 above
Read cost per feed load	1 range read	200 timeline queries + merge	1 range read + ~5 pulls + merge
Feed p99	Best	Worst — tail of 200 parallel calls	Close to push
Precomputed storage	~5 TB	None beyond timelines	~5 TB
Wasted work	High — writes for users who never read	None	Low
One celebrity post	~21 s of total cluster write capacity	Free; one hot cached key	Free
New follow visible	Only after a backfill job	Immediately	Immediately above threshold, backfill below
Complexity	Low	Low	Two paths, a threshold, merge logic

In an interview

What is tested is whether you spot the asymmetry — cheap frequent writes against latency-critical reads with wildly variable fan-out — and pick a strategy with arithmetic rather than preference. Underneath it is a hot-key question, and the same shape recurs in notification delivery and group chat broadcast.

Say it in this order:

1. State the fan-out multiplier before choosing anything: "1B posts a day times 200 average followers is 200B feed writes a day, about 2.4M per second."
2. Name both options and the end each one dies at: push dies on the celebrity, pull dies on a 200-way merge in the read path.
3. Propose the hybrid, give a threshold, and say it is tunable.
4. State what the feed store holds — IDs, capped at 500 — and why: content is mutable and would be duplicated 200 times.
5. Volunteer cursor pagination before you are asked. Interviewers use offset as a trap.

The mistakes that lose points:

• Choosing push and never mentioning the celebrity. This is what the question exists to test; a design that skips it reads as memorised.
• Choosing the hybrid without arithmetic. "Facebook uses a hybrid" is a recollection, not an answer. Deriving 2.4M writes/second and 21 seconds of whole-cluster capacity for one celebrity post is an answer.
• Storing post content in the feed. It says you have not considered edits, deletes, or memory cost.
• Offset pagination, and then being unable to explain what happens when a post arrives between page 1 and page 2.
• Folding ranking into fan-out. A score baked in at write time is a design you cannot iterate on.
• Treating the feed store as a source of truth. Say out loud that it is derived and rebuildable — that is what licenses the in-memory design, and why the feed API stays stateless.

If pushed on the threshold, do not defend 10,000 as a fact: "I would start at 10,000, watch the merge p99 and the fan-out queue lag, and move it. Having a threshold matters more than its value."

Check yourself

1. The product changes: average followers drops from 200 to 20, users post 10 times a day, and feeds are loaded 3 times a day. Still 500M DAU. Does push still win, and what actually changes?

Writes: 500M × 10 = 5B posts/day = 60,000 QPS. Fan-out: 5B × 20 = 100B feed writes/day ≈ 1.2M/second — half the original, so push is still affordable on the write side. But reads fall to 500M × 3 = 1.5B/day ≈ 18,000 QPS, a read-to-write ratio of 0.3:1: you are precomputing more feeds than you serve. Push pays its cost regardless of reads, so the wasted-work argument that killed celebrity push now applies to ordinary accounts. Push only into feeds of users active in the last 7 days, and rebuild lazily for the rest.

2. An account with 8,000 followers sits just under your 10,000 threshold and posts 40 times during a live event. What breaks, and is raising the threshold the fix?

40 × 8,000 = 320,000 feed writes, which over a day is ~4 writes/second and irrelevant. The problem is burst, not volume: 320,000 writes compressed into an hour, and what users see is queue lag — posts arriving minutes late — not throughput exhaustion. Raising the threshold makes it worse by admitting more accounts to push. The fixes are on the queue: partition it so one author cannot monopolise a worker pool, and make the threshold a rate as well as a count, so an account above N followers and M posts/hour drops to pull for the duration.

3. Estimate the memory cost of storing the author ID alongside each post ID in the feed, so unfollowed accounts can be filtered without a hydration round trip. Worth it?

Payload per entry goes from 8 bytes to 16, and with overhead from ~20 to ~28 bytes. 500 × 28 = 14 KB per user × 500M = 7 TB, up from 5 TB — a 40% increase, roughly 20 extra Redis nodes. Worth it only if unfollow-filtering is genuinely on the hot path, and it usually is not: you hydrate the posts anyway, and a hydrated post already carries its author. The rule it illustrates: denormalise into the feed only what you need before hydration.