Data Lakes & Warehouses
As organizations collect more data, they need places to store and analyze it at scale. The two dominant approaches — data warehouses and data lakes — make fundamentally different trade-offs. A new hybrid, the lakehouse, attempts to combine the best of both.
Data Warehouse
Structured, Schema-on-Write, Optimized for Analytics
- Schema-on-write: Data is cleaned and structured before loading
- SQL-based queries: Familiar interface for analysts and BI tools
- Columnar storage: Fast aggregations over specific columns
- Examples: Snowflake, BigQuery, Amazon Redshift
Trade-off: Fast queries, but rigid schema — changing structure is expensive.
Data Lake
Raw Storage, Schema-on-Read, Any Format
- Schema-on-read: Store raw data as-is, apply structure when querying
- Any format: JSON, CSV, Parquet, images, logs — all welcome
- Cheap storage: Object stores (S3, GCS) cost pennies per GB
- Examples: S3 + Athena, Azure Data Lake, Google Cloud Storage
Trade-off: Flexible and cheap, but queries are slower and data can become a "swamp" without governance.
The Lakehouse
Best of Both Worlds
A lakehouse adds warehouse-like features (ACID transactions, schema enforcement, indexing) on top of lake storage:
- Delta Lake: ACID transactions on Parquet files in S3
- Apache Iceberg: Table format with time-travel and schema evolution
- Apache Hudi: Incremental processing on data lake storage
You get the cheap, flexible storage of a lake with the query performance and governance of a warehouse.
ETL vs ELT
| Approach | Process | Best For |
|---|---|---|
| ETL | Extract → Transform → Load into warehouse | When transformation logic is complex and you want clean data in the warehouse |
| ELT | Extract → Load raw into lake/warehouse → Transform in-place | When compute is cheap (cloud), keep raw data, transform with SQL later |
Modern trend: ELT is winning because cloud warehouses have massive compute power and you preserve raw data for future use cases.
Star Schema & Dimensional Modeling
Warehouses typically use a star schema:
- Fact tables: Events/measurements (sales, clicks, transactions) — many rows, narrow
- Dimension tables: Context (users, products, dates, locations) — fewer rows, wide
- Facts reference dimensions via foreign keys, forming a star shape
This denormalized design trades storage for query speed — JOINs are predictable and fast.
Real-World Examples
Netflix: Lake + Warehouse Hybrid
Netflix processes petabytes of data daily:
- Data Lake (S3 + Iceberg): Stores raw event data — billions of playback events, A/B test results, device logs
- Processing (Spark + Flink): Transforms raw data into analytics-ready tables
- Query (Presto/Trino): Analysts query both raw lake data and processed tables with SQL
- Apache Iceberg gives them time-travel: "show me yesterday's version of this table"
The lake stores everything cheaply; Iceberg provides warehouse-like features without moving data.
Airbnb: Warehouse to Lakehouse Evolution
Airbnb's data platform evolved through three stages:
- V1 (Warehouse): Hive tables, batch ETL, slow iteration — days to add new data
- V2 (Lake): Raw data in S3, flexible schemas — but quality degraded ("data swamp")
- V3 (Lakehouse): Unified platform with certified datasets, quality checks, and discovery tools
Key lesson: a lake without governance becomes a swamp. Airbnb added data quality SLAs, lineage tracking, and a data catalog to make their lake useful.
Interactive: Classify Your Data
Where Should This Data Live?
For each data source, decide the best storage approach.