Real-Time Analytics Architecture

📘 Chapter 14: Data Pipelines & Stream Processing ⏱️ 9 min read 🏗️ Lesson 061

How do you aggregate and query millions of events per second with sub-second latency? You can't simply run SQL on a traditional database — you need a specialized architecture that pre-computes results and serves them from optimized storage.

The Real-Time Analytics Pipeline

Real-Time Analytics Pipeline Events Clicks, txns, IoT, logs Stream Processor Filter, aggregate, window, enrich Flink / Kafka Streams Materialized Views Pre-aggregated metrics & counters Druid / Pinot / ClickHouse Serving Layer API / Query engine Dashboard Sub-second refresh ~ms ~seconds pre-computed ~ms query End-to-end: event occurs → visible on dashboard in <5 seconds Key insight: compute BEFORE query time, not during
Figure 1: Real-time analytics pre-computes aggregations so queries are fast lookups, not expensive scans.

Pre-Aggregation: The Core Trick

Traditional analytics: user asks a question → scan billions of rows → return answer (slow).

Real-time analytics: continuously compute answers as data arrives → user asks → return pre-computed answer (fast).

You're trading storage and compute upfront for query speed at read time.

  • "Total orders in last 5 minutes" → maintained as a rolling counter
  • "Revenue by city today" → updated with each new order event
  • "p99 latency last hour" → computed from a streaming histogram

OLAP Engines

Columnar Storage for Fast Aggregation

Engine Strength Use Case
ClickHouse Fastest for ad-hoc analytical queries Log analytics, product analytics
Apache Druid Real-time ingestion + fast slice-and-dice Network monitoring, user-facing analytics
Apache Pinot Ultra-low latency on fresh data User-facing real-time dashboards at scale
StarRocks Unified batch + real-time analytics Replacing multiple engines with one

All use columnar storage — reading only needed columns makes aggregations 10–100x faster than row-based DBs.

Time-Series Databases

Optimized for Time-Indexed Data

When your primary query pattern is "show me metric X over time range Y":

  • InfluxDB: Purpose-built for metrics and IoT data
  • TimescaleDB: PostgreSQL extension — familiar SQL, time-series performance
  • Prometheus: Pull-based metrics for infrastructure monitoring

Optimizations: time-based partitioning, automatic downsampling (1-second → 1-minute → 1-hour), efficient compression of sequential timestamps.

Windowing Strategies

Stream aggregations need windows to bound the computation:

Window Type How It Works Example
Tumbling Fixed-size, non-overlapping Count orders every 5 minutes (0:00–0:05, 0:05–0:10…)
Sliding Fixed-size, overlapping Average latency over last 5 min, updated every 1 min
Session Gap-based, variable size Group user clicks until 30 min of inactivity

Real-World Examples

Uber: Apache Pinot for Surge Pricing

Uber uses Apache Pinot to power city-level real-time dashboards:

  • Ingests millions of trip events per second from riders and drivers
  • Computes supply/demand ratios per geo-cell in real time
  • Surge pricing decisions made on data that's <10 seconds old
  • Dashboards refresh every 2 seconds showing city-wide metrics
  • Handles 100,000+ queries per second with p99 latency <100ms

Without pre-aggregation, each dashboard refresh would scan billions of rows — impossible at this speed.

LinkedIn: "Who Viewed Your Profile"

LinkedIn's profile view feature processes billions of events daily:

  • Every profile view generates an event (viewer, viewed, timestamp, context)
  • Stream processor enriches with viewer's company, title, and connection degree
  • Pre-aggregates: "3 people from Google viewed you this week"
  • Results visible to users within seconds of the view happening
  • Historical trends ("your views are up 40% this month") computed via windowed aggregations

Interactive: Build a Real-Time Metrics Pipeline

Design Your Pipeline

Choose components for each layer and see the resulting latency and throughput characteristics.