Why Cache? The Memory Hierarchy
Every system you'll ever build has a fundamental bottleneck: the speed gap between where data lives and where it's needed. Caching exists because of a simple, brutal physical reality — moving data is slow, and the farther it has to travel, the slower it gets.
The Speed Gap
The memory hierarchy spans seven orders of magnitude in access time. Let's make this concrete:
Access Times at Each Level
- CPU Register — ~0.3ns (the speed of light across a chip)
- L1 Cache — ~1ns (on the CPU die)
- L2 Cache — ~4ns (still on-chip, larger)
- L3 Cache — ~10ns (shared across cores)
- RAM — ~100ns (off-chip, on the motherboard)
- SSD — ~100μs (100,000ns — a thousand times slower than RAM)
- HDD — ~10ms (10,000,000ns — mechanical seek)
- Network (same datacenter) — ~1ms
- Network (cross-continent) — ~100ms
The Caching Principle
Caching is simple: store frequently accessed data closer to where it's needed. That's it. Every caching system — from your CPU's L1 cache to a CDN — is applying this one idea at a different scale.
It works because of a property called locality of reference:
Why Caching Works
- Temporal locality — Data accessed recently is likely to be accessed again soon (e.g., a user's session, a trending post)
- Spatial locality — Data near recently accessed data is likely to be accessed next (e.g., sequential file reads, related DB rows)
Where Caching Happens in a Web System
In a modern web application, caching isn't a single layer — it's present at every stage of the request path:
The 80/20 Rule in Caching
Caching works spectacularly well because access patterns are almost never uniform. In most systems:
The Pareto Principle Applied
~20% of data serves ~80% of requests.
Think about it: on Reddit, the front page posts get millions of views while posts from 3 years ago get almost none. On Twitter, a tiny fraction of tweets (from high-follower accounts) generate most of the read traffic. On e-commerce sites, a small catalog of popular items dominates views.
This means you only need to cache a small subset of your data to get enormous performance gains.
Cache Hit Ratio: The Metric That Matters
The cache hit ratio = (cache hits) / (total requests). It tells you how effective your cache is:
- 95%+ hit ratio — Excellent. Your cache is absorbing nearly all traffic.
- 80-95% — Good. Most reads bypass the origin.
- Below 80% — Investigate. Either your data isn't cacheable, your TTL is too short, or your cache is too small.
When Caching Isn't Worth It
- Data that changes on every request (real-time stock prices at millisecond granularity)
- Highly personalized data with no overlap between users
- Write-heavy workloads where cache is invalidated faster than it's read
- When the origin is already fast enough (adding cache adds complexity)
Real-World: How Reddit Caches Hot Posts
🏢 Reddit's Caching Architecture
Reddit's front page serves millions of requests per second. Almost none of them hit the database.
- The front page listing is pre-computed and stored in Memcached. It's regenerated periodically (every few seconds) rather than on each request.
- Hot posts (top ~1000 across all subreddits) are cached in memory. This tiny subset serves the vast majority of page views.
- Comment trees for popular posts are cached as serialized blobs — rendering the tree from DB on every view would be impossibly expensive.
- Vote counts are cached and updated asynchronously — the number you see might be a few seconds stale, but that's acceptable for the performance gain.
The result: Reddit can serve its most popular content with sub-millisecond response times, even during traffic spikes.
Interactive: Explore the Memory Hierarchy
Click each layer to see access times, typical sizes, and real-world examples:
Click a layer above to explore it.