Chapter 14 Interview: Data Pipelines

It really comes down to latency requirements and the nature of the computation. Let me give you my mental model for choosing.

Batch is your workhorse when you need to process large volumes and the results don't need to be immediate. Think nightly reports, training ML models, rebuilding search indexes, generating invoices at end-of-month. You collect data over a period, then crunch it all at once. Tools like Spark, Hadoop MapReduce — they're optimized for throughput over latency.

Stream processing is for when you need to react in near-real-time. Fraud detection — you can't wait until tomorrow to flag a suspicious transaction. Monitoring and alerting, live dashboards, real-time recommendations. Tools like Kafka Streams, Flink, or Spark Structured Streaming process events as they arrive, typically within milliseconds to seconds.

Here's the nuance though — many systems use both. The Lambda architecture has a batch layer for correctness and a speed layer for low-latency approximations. More modern systems lean toward Kappa architecture — just stream everything, and if you need historical reprocessing, replay the event log. I'd default to streaming if latency matters and batch if it's purely analytical workloads where throughput and cost efficiency win.

• Batch: high throughput, hours-old data acceptable, complex aggregations
• Stream: low latency, event-at-a-time, real-time reactions needed
• Lambda: both layers — batch for accuracy, stream for speed
• Kappa: stream-only with replay capability for reprocessing

So event sourcing flips the usual database model on its head. Instead of storing current state — like "account balance is $500" — you store the sequence of events that led to that state: "deposited $1000, withdrew $300, transferred $200." The current state is derived by replaying events.

Think of it like a git history vs just having the current files. You can always reconstruct any past state by replaying events up to that point. That's incredibly powerful for auditing — financial systems, healthcare, legal compliance — anywhere you need a complete, immutable history of what happened.

It also enables temporal queries — "what was this account's balance on March 3rd?" — and makes debugging easier because you can replay events to reproduce bugs. Plus you get natural event-driven integration: other services can subscribe to the event stream and build their own views.

When does it NOT make sense? Simple CRUD apps where you just need current state and history doesn't matter. The complexity cost is real — you need event stores, projection builders, snapshot strategies for performance. If your domain doesn't benefit from the audit trail or temporal queries, you're paying that cost for nothing.

• Core idea: store events (facts), derive state by replay
• Benefits: complete audit trail, temporal queries, event-driven integration
• Good fit: financial systems, compliance-heavy domains, collaborative editing
• Bad fit: simple CRUD, high-volume writes with no audit need
• Complexity: snapshots for performance, eventual consistency, schema evolution

Okay so the key challenge is: dashboards need fast reads across potentially billions of events, updated in near-real-time. You can't just query your OLTP database — it'll melt. Here's how I'd architect it.

Events flow into Kafka as the central nervous system. From there, a stream processor — Flink or Kafka Streams — computes pre-aggregated metrics in real time. Things like "requests per minute by endpoint" or "revenue in the last hour by region." These aggregations get written to a fast read store — something like Apache Druid, ClickHouse, or even Redis for the hottest metrics.

The dashboard frontend polls or subscribes via WebSocket for updates. For real-time feel, I'd use WebSockets to push updated aggregations every few seconds rather than having the client poll. Server-Sent Events work too if it's one-directional.

Important design choice: pre-aggregate at write time, not query time. If you try to compute "average response time over the last hour" by scanning raw events on every dashboard load, you're dead at scale. The stream processor maintains running windows and emits results continuously.

For historical drill-down — "show me last month" — you'd have a separate batch-computed layer in something like a columnar data warehouse. The real-time layer handles the last few hours; the batch layer handles everything before that.

• Ingest: Kafka for durable, ordered event capture
• Process: Flink/Kafka Streams for windowed pre-aggregation
• Serve: Druid/ClickHouse/Redis for low-latency dashboard reads
• Deliver: WebSockets for push-based real-time updates to the UI

They solve different problems and honestly, most mature companies end up with both. But let me explain the trade-offs.

A data warehouse — Snowflake, BigQuery, Redshift — stores structured, processed data optimized for analytical queries. Schema-on-write: you define the schema upfront, transform data on the way in (ETL), and analysts get clean, fast SQL queries. Great for business intelligence, dashboards, known questions.

A data lake — S3/GCS with something like Delta Lake or Iceberg — stores raw data in any format: JSON, Parquet, images, logs, whatever. Schema-on-read: you dump everything in and figure out structure when you query it. Great for data science exploration, ML training data, archiving everything cheaply in case you need it later.

My decision framework: if I know the questions I'm asking and need fast, repeatable analytics — warehouse. If I'm exploring, doing ML, or need to store diverse data cheaply without knowing future use cases — lake. Cost is a factor too: lakes on object storage are 10-100x cheaper per TB than warehouses.

The modern trend is the "lakehouse" — Delta Lake, Iceberg — that adds warehouse-like features (ACID transactions, schema enforcement, fast queries) on top of lake storage. Best of both worlds, and what I'd default to for new greenfield systems.

• Warehouse: structured, schema-on-write, fast SQL, BI-focused
• Lake: raw, schema-on-read, cheap storage, ML/exploration
• Cost: lakes are dramatically cheaper per TB
• Lakehouse: modern hybrid — lake storage + warehouse query performance

Late data is inevitable in distributed systems. A mobile device goes offline, a network partition delays events, a batch upload arrives hours after the fact. You have to design for it, not pretend it won't happen.

The core concept is watermarks. A watermark is the system's estimate of "I believe I've seen all events up to time T." Events arriving after the watermark are considered late. In Flink, you configure how long to wait past the watermark before closing a window — that's your allowed lateness.

For windowed aggregations, you have options. You can hold windows open longer (higher latency but more complete). You can emit early results and then update them when late data arrives — this is what Flink's "allowed lateness" does, re-firing the window computation. Or you can route late data to a side output for separate handling.

At the storage layer, I'd design for upserts rather than append-only. If a daily aggregation gets a late event, you update that day's total. Idempotent processing helps here — reprocessing the same event shouldn't double-count it. Use event IDs for deduplication.

The business determines the acceptable trade-off. Real-time fraud detection can't wait — accept incompleteness. Monthly billing must be exact — hold the window open longer or do reconciliation passes.

• Watermarks: system's estimate of event-time completeness
• Allowed lateness: configure windows to accept and re-process late events
• Side outputs: route extremely late data for special handling
• Idempotent upserts: design aggregations to handle reprocessing safely