Batch vs Stream Processing

📘 Chapter 14: Data Pipelines & Stream Processing ⏱️ 8 min read 🏗️ Lesson 058

Every data system must answer: when do we process data? All at once after it accumulates (batch), or continuously as it arrives (stream)? This fundamental choice shapes your architecture's latency, throughput, and complexity.

Batch Processing

Collect → Process → Output

Batch processing accumulates data over a period, then processes it all together. Think MapReduce, Apache Spark, nightly ETL jobs.

  • High throughput: Process millions of records efficiently
  • High latency: Results are hours or days old
  • Simple error handling: Rerun the whole job if it fails
  • Complete data: All records available before processing starts

Stream Processing

Process Each Event as It Arrives

Stream processing handles data continuously — event by event or in micro-batches. Think Kafka Streams, Apache Flink, AWS Kinesis.

  • Low latency: Results in seconds or milliseconds
  • Lower throughput per event: Overhead of processing each record individually
  • Complex error handling: Must handle out-of-order, late, and duplicate events
  • Incomplete data: You never have "all" the data — it keeps arriving
Batch vs Stream Processing Batch Processing Collect Hours/Days Process All MapReduce/Spark Output Results ready ⏱️ Latency: hours | 🚀 Throughput: very high Stream Processing Process Results emitted continuously ⏱️ Latency: ms–seconds | 🚀 Throughput: moderate per event Stream Windowing Window 1 Window 2 Window 3 Group events into time windows for aggregation
Figure 1: Batch collects everything then processes; stream processes continuously with windowing for aggregation.

Lambda Architecture

Batch + Stream in Parallel

The Lambda architecture runs both a batch layer and a speed layer in parallel:

  • Batch layer: Processes all historical data for accuracy (eventual, complete results)
  • Speed layer: Processes recent data for freshness (approximate, real-time results)
  • Serving layer: Merges both views for queries

Downside: You maintain two codebases (batch logic + stream logic) that must produce the same results.

Kappa Architecture

Stream-Only (Replay When Needed)

The Kappa architecture simplifies Lambda by using only stream processing:

  • All data flows through a single stream pipeline
  • For reprocessing, replay events from an immutable log (e.g., Kafka)
  • One codebase, simpler to maintain

Downside: Replaying years of history through a stream processor can be slow.

When to Use Each

Use Case Best Fit Why
Nightly reports Batch All data available, no urgency
ML model training Batch Needs complete dataset, high compute
ETL to warehouse Batch Scheduled transforms, throughput matters
Fraud detection Stream Must act in milliseconds
Real-time dashboards Stream Users expect live data
Push notifications Stream Timely delivery is the whole point

Real-World Examples

Spotify: Batch + Stream

Spotify uses both processing models:

  • Batch (daily): Discover Weekly playlists — runs overnight using weeks of listening history, collaborative filtering on billions of data points, delivered every Monday
  • Stream (real-time): "Friend Activity" sidebar — shows what friends are listening to right now, processed event-by-event with sub-second latency

Batch gives Spotify the deep analysis needed for personalization; stream gives the liveness users expect from a social music platform.

Bank Fraud Detection with Stream Processing

A major bank processes 50,000 transactions per second through Apache Flink:

  • Each transaction is evaluated against rules and ML models in <100ms
  • Windowed aggregations detect patterns: "5 transactions in different countries within 10 minutes"
  • Suspicious transactions are flagged before the merchant even confirms the charge
  • Batch runs nightly to retrain fraud models on the full day's data

Stream catches fraud in real time; batch improves the models that stream uses.

Interactive: Batch or Stream?

Choose the Right Processing Model

For each scenario, decide: batch, stream, or both?