Batch vs Stream
Choose batch or stream from a staleness number, size windows and partitions, and stop at-least-once duplicates from inflating every aggregate.
The choice between batch and stream is a freshness requirement with a number attached, not a technology preference. Batch waits for a bounded chunk and computes over all of it; stream computes over each event as it arrives and never sees all of it.
Batch processing
Batch processing collects data over a period and processes the whole batch in one job — a week of clothes, one Sunday load.
- High throughput. Sequential reads, no per-event overhead, and failure handling as coarse as running the job again.
- High latency. Results arrive on the schedule, not on the event: an event landing at 02:05 waits ~24 hours for the next 02:00 run.
- Bounded data. A finite, static input, which is what lets the job be exact: late arrivals are already in the file, and joins cross the whole history.
Use it for end-of-day financial reports, ETL into a warehouse, deep analytics over history, and model training. Hadoop MapReduce was the original distributed batch framework; Spark superseded it by keeping intermediate results in memory rather than spilling each stage to disk.
The failure mode nobody schedules for
The batch failure that reaches production is not a crash, it is the schedule closing on itself — an hourly job that takes 70 minutes:
run interval 60 min
run duration 70 min
slip per cycle 10 min
after 24 runs 24 × 10 = 240 min = 4 hours behind, and growing
Nothing throws. The dashboard just gets older until someone notices the numbers are from lunchtime. A schedule is a queue, and once service time exceeds the interval, utilisation passes 1 and the backlog grows without bound — a stability condition, not Little's Law, which describes a queue that is already stable. The fix is an interval with headroom over the p99 run time, or no interval at all.
Stream processing
Stream processing handles each event as it arrives — the dish washed as soon as the meal ends.
- Low latency. Results within milliseconds or seconds of arrival, so the system can react while the event still matters.
- Unbounded data. No defined end, so "the input is complete" never happens and the job computes a window rather than a total.
Use it for fraud checks on live transactions, clickstream monitoring, live dashboards, and anomaly alerting. Flink processes event-at-a-time with managed state; Spark Structured Streaming reaches similar ground through micro-batches; Kafka Streams is a client library reading straight from topics, enough when the job belongs to one service and its own event log.
Windows, watermarks, and what they cost
A window has to close before it can emit, and deciding when is the hard part. Wall clock is the wrong answer: a phone in a tunnel sends its 12:03 events at 12:09. The gap between event time and processing time is where most stream bugs live.
A watermark is the job's claim that it has probably seen everything up to event time T. Aggressive watermarks close windows fast and drop late data; conservative ones improve completeness and make every result wait. No setting gets both, so set it from the requirement, not the default.
State is the second cost. A per-user aggregate over 10 million active users at 200 bytes each is 2 GB of live state to checkpoint. On restart the job restores that checkpoint and resumes from the stored offset, so every event already processed after that offset is processed again.
At-least-once, duplicates, idempotency
Restarts, rebalances and redeploys all replay from the last committed offset, which makes a stream pipeline at-least-once by default; duplicates inside an aggregation drift the number upward on every deploy. The sink has to be idempotent: an upsert keyed by window and key rather than an increment, a dedupe on event id, or a transactional sink that commits output and offsets together. "Exactly-once" in a vendor's documentation names that machinery; it never means duplicates stopped arriving.
Choosing between them
The decision reduces to one question with a number in it: how stale can this answer be before it stops being worth computing? Above an hour, batch is simpler, cheaper and more accurate, and a queue feeding a scheduled job is the whole design. Under a minute, no schedule reaches the target. Between the two, the tiebreak is correctness rather than cost.
Running both over the same events
Many systems run both: a stream job for the live view, a batch job over the same retained log for the exact one. That is the Lambda architecture, and its price is two implementations of one business rule that must agree — every change is two changes and a reconciliation. Kappa replays the retained log through the same streaming code into a new output table: one codebase, paid for in retention and replay time. Take Lambda only when the batch layer does something streaming cannot: a full-history join, or a model too expensive to run per event.
In an interview
What is being tested is whether a freshness number drives the design, not whether you can name Flink. The requirement is usually planted in a soft sentence — "roughly live", "block the fraudulent charge" — and the interviewer waits to see whether you turn it into a bound before choosing.
Phrasing that scores: "Settlement is reconciled daily, so the report is a nightly Spark job — batch, because late corrections are already in the file. Blocking a fraudulent authorisation happens inside a 200 ms path, so a Flink job keeps the per-card counter and the request path reads it as one lookup. Same events, two consumers, two latency budgets." Then price it unprompted: the stream side is at-least-once, so that counter is an upsert keyed by card and window.
The mistake that loses the most points is naming Kafka and Flink before stating the staleness bound that rules out a scheduled job. It reads as pattern-matching rather than design, and it invites the follow-up most candidates miss: what does your stream get wrong? Late events and duplicates.
Check yourself
1. A clickstream runs at 1 billion events/day, 1 KB each, the dashboard must be under 5 minutes stale, and the hourly Spark job takes 70 minutes. What do the numbers rule out?
1 billion/day ≈ 12,000 events/s average, ~36,000/s at 3x peak, 1 TB/day. The freshness target kills the design on its own: no hourly schedule produces an answer under 5 minutes old, whatever the run time. The second problem is silent — 70 minutes of work on a 60-minute schedule slips 10 minutes per cycle, so 4 hours of lag accumulate per day, without bound. Stream the dashboard; give any surviving batch job headroom over its p99 run time.
2. A fraud decision must land inside a 200 ms p99 authorisation path, and the rule needs "transactions on this card in the last 10 minutes". Does a windowed stream job answer the request?
No. The network is not the constraint — a datacenter round trip is 0.5 ms — the window is: a 10-minute window emits when it closes, up to 10 minutes after the event that mattered. Split the paths: the stream job keeps the per-card counter in state and publishes it to a fast store; the synchronous path does one lookup, on the order of a millisecond. Ruled out: any design where the request waits on a window emit.
3. Peak is 36,000 events/s and one consumer instance handles about 2,000/s. How many partitions, and what has to hold for the sink after a redeploy?
36,000 ÷ 2,000 = 18 consumers minimum, so 24 partitions leaves headroom for growth and for a rebalance that loses an instance. Partition count is awkward to reduce later and ordering holds only within a partition, so key by the entity being aggregated, not round-robin. After a redeploy the job resumes from the last committed offset and reprocesses what it had already handled past it, so the sink must be an idempotent upsert keyed by window and key; an incrementing counter would climb on every deploy.