Rate Limiting & Throttling

📘 Chapter 7: Scalability Patterns ⏱️ 10 min read 🏗️ Lesson 030

Rate limiting is both a protective mechanism (prevent abuse, stop cascade failures) and a fairness mechanism (ensure one noisy client can't starve everyone else). Every production API needs it.

Why Rate Limit?

  • Prevent abuse: Stop brute-force attacks, credential stuffing, web scraping at scale
  • Ensure fair usage: One customer shouldn't consume 90% of capacity
  • Prevent cascade failures: A buggy client sending 10K req/s can take down your service for everyone
  • Cost control: Especially for paid APIs or services with metered downstream dependencies
  • Compliance: Some regulations require demonstrating you can throttle abusive usage

Algorithms: Token Bucket vs Leaky Bucket

Token Bucket vs Leaky Bucket Token Bucket capacity = 10 tokens ↓ refill: 2 tokens/sec request takes 1 token • Allows bursts (up to bucket size) • Steady rate = refill rate • Empty bucket → reject (429) Used by: AWS, Stripe, most APIs Leaky Bucket req 5 (queued) req 4 (queued) req 3 (queued) req 2 (queued) req 1 (processing) ↓ drains at constant rate: 2/sec overflow → reject • Smooths out bursts (constant output) • Requests queue up in the bucket • Full queue → reject (429) Used by: Nginx, traffic shaping
Figure 1: Token Bucket allows bursts then limits; Leaky Bucket enforces a constant output rate by queuing requests.

Window-Based Algorithms

Fixed Window vs Sliding Window Fixed Window (limit: 10/min) time → Window 1 (00:00-01:00) 4 requests Window 2 (01:00-02:00) 6 requests ⚠️ Boundary: 10 req in 30s span! Sliding Window (limit: 10/min) time → ← 60s sliding window → 8 requests in any 60s window ✓ Window slides with each request — no boundary exploit Simple counter per window. Reset at boundary. Bug: 2× burst possible at window edges. Weighted count using previous + current window. No boundary exploit. Slightly more memory.
Figure 2: Fixed windows allow boundary exploits (2× burst). Sliding windows eliminate this by considering a rolling time span.

Where to Implement Rate Limiting

  • API Gateway / Edge: First line of defense. Blocks abusive traffic before it reaches your services. (Kong, AWS API Gateway, Cloudflare)
  • Reverse Proxy: Nginx limit_req module — fast, lightweight, per-IP or per-route
  • Application Layer: Business-logic aware — rate limit per user, per API key, per plan tier
  • Per-Service: Each microservice protects itself — defense in depth

Best practice: Layer them. Edge catches volume attacks; application layer enforces business rules.

HTTP Headers for Rate Limits

Communicate Limits Clearly

Standard response headers (draft RFC 6585 / RateLimit fields):

  • X-RateLimit-Limit: 5000 — max requests allowed in the window
  • X-RateLimit-Remaining: 4992 — requests left in current window
  • X-RateLimit-Reset: 1700000000 — Unix timestamp when the window resets
  • Retry-After: 30 — seconds to wait before retrying (on 429 responses)

When rate limited, return HTTP 429 Too Many Requests with a clear error body explaining what happened and when to retry.

Distributed Rate Limiting

The Multi-Server Challenge

With 10 servers, each counting locally, a client gets 10× the intended limit. Solutions:

  • Redis-based counters: All servers increment the same key in Redis. Use INCR + EXPIRE for fixed windows.
  • Lua scripts for atomicity: Redis Lua scripts execute INCR + check + EXPIRE in one atomic operation — no race conditions.
  • Sliding window in Redis: Use sorted sets — ZADD with timestamp, ZRANGEBYSCORE to count requests in the window, ZREMRANGEBYSCORE to clean up.
  • Approximate (local + sync): Each server tracks locally, periodically syncs to a central store. Faster but allows slight over-limit.

🏢 Real-World: GitHub's API Rate Limiting

  • Authenticated: 5,000 requests per hour per user (identified by OAuth token)
  • Unauthenticated: 60 requests per hour per IP address
  • Search API: Separate, stricter limit — 30 requests per minute (expensive queries)
  • Headers: Every response includes X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset
  • Conditional requests: Requests returning 304 Not Modified don't count against limits — encourages caching
  • Abuse detection: Beyond rate limits, GitHub detects patterns (rapid sequential page access) and temporarily blocks with 403

🏢 Real-World: Stripe's Graceful Rate Limiting

  • Per-key limits: Rate limits are per API key, not per IP — fairer for customers behind NAT
  • Graduated response: First, slow down (add latency). Then reject with 429. Only ban for sustained abuse.
  • Clear error messages: The 429 response body explains exactly which limit was hit, current usage, and when to retry
  • Load shedding: Under extreme load, Stripe returns 503 with Retry-After — admits overload rather than serving errors
  • Test mode: Higher limits in test mode so developers can iterate fast without hitting walls

Interactive: Token Bucket Visualizer

Configure the token bucket and send requests to see tokens deplete and refill:

Tokens Available
10
0 10
Request Log:
Accepted
0
Rejected
0