Kafka vs RabbitMQ
Choose between a routing broker and a durable log from replay, retry and throughput needs, then size partitions, retention and consumers before naming either.
RabbitMQ and Apache Kafka both move messages between services and are routinely offered as alternatives to each other. They are not interchangeable: one is a router that forgets, the other is a log that remembers. Deciding you want a message queue settles that a call should be asynchronous, not which of these two shapes fits — and the wrong pick surfaces months later as a replay you cannot perform or a retry you cannot express.
RabbitMQ: the smart message broker
Analogy: a smart post office. It receives messages (letters), understands routing rules (zip codes, recipient names, mail types), and delivers them to specific mailboxes (queues). The broker does the work.
Architecture: a message broker implementing protocols such as AMQP. Producers send to an exchange, which routes to one or more queues by rule — a direct match on a routing key, a topic pattern, a fanout to every bound queue. Consumers listen on queues.
Key feature: the broker tracks the delivery state of each message. A message is removed once a consumer acknowledges it; a rejected one goes back for redelivery, or to a dead-letter exchange after a retry limit. Per-message state is the product.
Apache Kafka: the distributed streaming platform
Analogy: a durable, append-only logbook. Producers append records to the end. Consumers read the log and keep a bookmark — an offset — of how far they have read.
Architecture: a distributed, partitioned, replicated commit log. A topic splits into partitions; each partition has a leader broker taking writes and followers replicating it, and acks=all waits for the in-sync followers before acknowledging. The broker does not track what a consumer has read.
Key feature: the broker is dumb (it stores the log), the consumers are smart (they hold their own position). Records are retained for a configured period whether or not anyone consumed them, so several consumer groups read one stream independently and any of them can rewind and replay.
Comparison
| Feature | RabbitMQ | Apache Kafka |
|---|---|---|
| Architecture | Smart broker (complex routing) | Distributed streaming platform (durable log) |
| Primary use case | Traditional message queuing, background jobs, task distribution | High-throughput streaming, real-time data pipelines, event sourcing |
| Message model | Point-to-point and publish/subscribe via exchanges and queues | Publish/subscribe via topics and partitions |
| Message delivery | Push-based: broker pushes to consumers | Pull-based: consumers pull from the broker |
| Retention | Deleted after consumption and acknowledgement | Retained for a configured period, days or more |
| Throughput | Order of 10,000–20,000 msg/s per node durable and acked, roughly twice that for transient classic queues | Order of hundreds of thousands per second, scaled by adding partitions and brokers |
| Ordering | FIFO into a queue, but preserved through processing only with one consumer and no requeues | Guaranteed only within a single partition |
| Consumers | Compete for messages on one queue | A consumer group divides partitions among its members |
What the log costs
Retention is disk, so do the arithmetic before promising a week of replay. A clickstream at 20M DAU and 30 events per user per day is 600M events/day ÷ 86,400 s ≈ 7,000 events/s average, ~21,000/s at 3x peak. At 1 KB per event that is 600 GB/day, and seven days at replication factor 3 is 600 GB × 7 × 3 ≈ 12.6 TB before overhead — making "keep 30 days, replay is useful" a 54 TB decision rather than a config flag.
Partitions set the parallelism ceiling. A consumer group can have at most one consumer per partition; extra members sit idle. If one consumer sustains 2,000 events/s, an 8-partition topic tops out at 16,000/s and cannot absorb the 21,000/s peak however many pods you add — lag grows until retention expires and data is lost. You need 21,000 ÷ 2,000 ≈ 11, so provision 12 or more up front, because the later fix has its own failure mode: adding partitions remaps keys, so a user whose events landed on partition 3 starts landing on partition 9 while older events still sit unconsumed on 3, and per-key ordering breaks across the change.
Keying by tenant or user has a second trap. One hot key — a large merchant, a celebrity account — pins its whole stream to one partition, and that consumer becomes the bottleneck while eleven others idle. The fix is a compound key, paid for with the ordering guarantee you picked the key for.
What the smart broker costs
RabbitMQ gives what Kafka does not: per-message fate. One message can be nacked, retried with backoff, and dead-lettered after five failures while everything behind it keeps flowing. In Kafka a poisoned record blocks its partition, because offsets advance in order — you skip it and lose it, or build retry topics yourself.
The cost is that a backlog becomes the broker's problem rather than a number on a disk. RabbitMQ holds queue state in memory; a queue filling at 5,000 msg/s of 1 KB messages grows 5 MB/s, about 18 GB over an hour of consumer downtime. Past the memory or disk alarm the broker applies flow control and blocks publishers, so producer latency rises and the decoupling the queue existed to provide is gone at the moment you needed it. Kafka meets the same hour with sequential appends and a larger offset lag, and nothing about the broker changes. Quorum queues raise RabbitMQ's durability by replicating through Raft, paying a majority round trip per publish — see distributed consensus for what that quorum costs.
Delivery guarantees are the same story on both
Both default to at-least-once: a consumer that processes a message and dies before acknowledging (RabbitMQ) or before committing its offset (Kafka) sees that message again. At-least-once means duplicates, and duplicates mean the consumer must be idempotent — a dedupe key, a unique constraint, an upsert. Kafka's exactly-once semantics do not remove that: they make the read-process-write cycle inside Kafka atomic by committing offsets in the same transaction as the output records. The email you send and the card you charge sit outside that transaction and can still happen twice.
Publishing from a service that also writes a database carries a separate hazard: the commit and the publish are two systems with no transaction spanning them. That is the dual-write problem and the outbox pattern, and it applies to both brokers alike.
Choosing
RabbitMQ when routing is real, when the work is background jobs needing per-message retry and dead-lettering, and when throughput sits in the thousands to roughly ten thousand messages per second, inside one node's durable, acked ceiling.
Kafka when several independent consumer groups need one stream, when replay is a requirement rather than a nicety, when volume runs past roughly 20,000 per second, or when the events are the source of truth. If the pipeline feeds analytics, read batch vs stream first.
Plenty of systems run both, and saying so beats forcing one tool onto both jobs.
In an interview
What is being tested is whether you pick infrastructure from the data flow rather than from reputation. Name the deciding property in one sentence: "several consumer groups need this stream and analytics must replay a week, so Kafka," or "this is one background job with retries and a dead-letter queue at 7 jobs per second, so RabbitMQ." Then price it — partitions from peak throughput divided by per-consumer rate, retention from events per day times payload times replication factor.
Show they are not interchangeable by using both in one design: "welcome emails go through RabbitMQ, per-message retry at trivial volume; the clickstream goes to Kafka so the live dashboard and the nightly batch job read one stream at their own offsets."
The mistake that loses points is naming Kafka for every asynchronous need, then having no answer for how one poisoned message is retried without stalling its partition. Next is claiming Kafka gives exactly-once delivery end to end; it gives exactly-once processing within Kafka, and the external side effect still needs idempotency. Third is buying a broker at a volume that does not need one — a jobs table with SELECT … FOR UPDATE SKIP LOCKED in the Postgres you already run beats a new operational component at single-digit QPS.
Check yourself
1. A topic takes 600M events/day, each consumer sustains 2,000 events/s, and peak is 3x average. How many partitions, and what breaks if you start with 8 and add more later?
600M ÷ 86,400 ≈ 7,000 events/s average, ~21,000/s at peak. 21,000 ÷ 2,000 ≈ 11 consumers, so at least 12 partitions and more for headroom, since a group cannot exceed one consumer per partition. Starting at 8 caps throughput at 16,000/s and lag grows until retention expires. Adding partitions later remaps keys, so a key's new events land on a different partition from its unconsumed old ones and per-key ordering breaks across the change.
2. 200,000 background jobs per day, each needing retry with backoff and a dead-letter after five failures, routed by job type. Kafka or RabbitMQ, and what rules the other one out?
200,000 ÷ 86,400 ≈ 2.3 jobs/s average, under 10/s at peak, so throughput decides nothing. The requirements are per-message fate and routing, which is RabbitMQ's model directly. Kafka is ruled out by retry semantics, not volume: a failing job blocks its partition unless you hand-build retry topics. At this rate, name the alternative you rejected too — a jobs table with SKIP LOCKED in the existing database, defensible until routing rules or throughput grow.
3. You are asked for 30 days of replay on a 1 KB event stream at 7,000 events/s with replication factor 3. What is the storage bill, and what do you propose instead?
7,000 × 86,400 ≈ 600M events/day × 1 KB = 600 GB/day. 30 days × 3 replicas = 54 TB, against 12.6 TB for seven days. Keep the hot log short — three to seven days, enough for consumer recovery and reprocessing — and archive to object or tiered storage for the long tail, since 30-day replay is a backfill job rather than a broker requirement. Reconsider only if the events are the system of record, which makes retention a durability decision instead of a convenience.