Design a News Feed (End-to-End)

📘 Chapter 15: Putting It All Together ⏱️ 9 min read 🏗️ Lesson 064

The news feed problem boils down to one fundamental trade-off: do you pay the cost at write time (push) or read time (pull)? This single decision shapes the entire architecture.

Requirements

Functional

  • Users see a personalized feed of posts from people they follow
  • Feed updates in near-real-time when someone posts
  • Support text, images, links, reactions
  • Ranked by relevance (not just chronological)

Non-Functional

  • Scale: 500M users, 10M posts/day, avg 300 followers
  • Latency: Feed loads in <500ms
  • Freshness: New posts appear within 5 seconds
  • Availability: 99.99% — feed is the core product

The Fan-Out Problem

Push vs Pull Fan-Out Push Model (Write-time) User Posts Fan-out Service Feed A Feed B Feed C Feed N… ✓ Fast reads (pre-built) ✗ Expensive writes (N copies) ✗ Celebrity problem (10M followers) Pull Model (Read-time) Posts X Posts Y Posts Z Posts N… Merge + Rank at read time User Feed ✓ Simple writes (just store post)
Figure 1: Push pre-computes feeds at write time; Pull aggregates at read time. The celebrity problem makes pure push impractical.

The Hybrid Approach

The solution used by Twitter and Facebook:

  • Normal users (≤10K followers): Push — fan-out on write to followers' feed caches
  • Celebrities (>10K followers): Pull — their posts are fetched at read time and merged

When a user opens their feed: fetch pre-built feed (push results) + merge in celebrity posts (pull results) + rank everything.

Why? A celebrity with 10M followers posting once = 10M writes. That's too expensive. Instead, merge their posts only when a follower actually reads their feed.

Full Architecture

Post Service Create post Fan-out Workers (async) Feed Cache Redis (per-user) Feed Service Merge + Rank Client Render feed Celebrity Posts (pulled at read time) Ranking ML model: engagement, recency, affinity WRITE PATH READ PATH
Figure 2: Hybrid architecture — fan-out on write for normal users, pull at read time for celebrities, ML ranking on top.

Feed Ranking

From Chronological to ML-Based

  • Simple: Reverse chronological — newest first (early Twitter)
  • Engagement: Score = likes × 2 + comments × 3 + shares × 5 + recency_decay
  • ML: Thousands of signals — relationship strength, content type preference, time-of-day patterns, interaction history

Facebook's ranking considers 10,000+ signals and runs inference on every feed load. The ranking model alone is a team of 100+ engineers.

Twitter: Pull → Push → Hybrid

  • Early Twitter: Pure pull — query all followed users' tweets at read time. Slow at scale.
  • 2012 redesign: Moved to push (fan-out on write). Each tweet fanned out to all followers' timelines in Redis.
  • Celebrity problem: Lady Gaga tweeting = 30M+ writes. Solution: hybrid — celebrities are pulled at read time.
  • Timeline cache: sorted set in Redis, capped at 800 tweets per user

Facebook News Feed: ML at Extreme Scale

  • Average user has 1,500+ candidate stories per feed load
  • Multi-pass ranking: lightweight filter → candidate scoring → final ranking
  • Features include: post type, poster affinity, content freshness, predicted engagement
  • A/B tests ranking changes on millions of users before rolling out

Interactive: Fan-Out Cost Calculator

Calculate Fan-Out Cost

Adjust parameters to see how push vs pull costs compare.


10M

300

10K

1%