Chapter 7 Interview: Scalability

Okay so honestly, the first thing I'd do is NOT immediately throw more servers at it. I'd start by profiling — where's the actual bottleneck? Is it CPU-bound computation, database queries, network I/O? Because 10x traffic doesn't mean you need 10x of everything.

The way I see it, you attack this in layers. First, caching — a well-placed cache can absorb 80% of read traffic. Redis or Memcached in front of your database, CDN for static assets, maybe even application-level memoization. That alone might get you 3-5x.

Next, horizontal scaling of the stateless services. If your app servers are stateless (and they should be), you just add more behind the load balancer. Auto-scaling groups that react to CPU or request queue depth.

Then the database — that's usually the real wall. Read replicas first, then if writes are the bottleneck, you're looking at sharding or moving to a distributed database. But I'd exhaust simpler options first — query optimization, connection pooling, read replicas with smart routing.

And honestly? I'd also look at what work can be moved async. Do you really need to process everything in the request path? Push stuff to queues and handle it in background workers.

• Profile first: identify the actual bottleneck before scaling blindly
• Caching: CDN, Redis, app-level — absorbs most read traffic
• Horizontal scaling: stateless services behind auto-scaling LBs
• Database: read replicas → sharding → distributed DB (exhaust simple first)
• Async offload: move non-critical work out of the request path

So basically, rate limiting is about protecting your system from being overwhelmed — whether that's malicious attacks, buggy clients, or just unexpected traffic spikes. I'd think about it at multiple levels.

For the algorithm, I'd probably go with token bucket or sliding window. Token bucket is nice because it allows short bursts while enforcing an average rate. Like, "100 requests per minute, but you can burst up to 20 in a second if you haven't been active." Sliding window log is more precise but uses more memory.

Where to enforce it — ideally at the API gateway or reverse proxy layer, before requests even hit your application servers. That way you're rejecting cheap, at the edge. But you also want application-level limits for more granular control — per-user, per-endpoint, per-action.

For distributed rate limiting across multiple servers, you need a shared store. Redis is the go-to — INCR with TTL for simple counters, or Lua scripts for token bucket logic atomically. The key is making the check fast — you can't add 50ms to every request for a rate limit check.

Response-wise, return 429 Too Many Requests with a Retry-After header. Be nice to your clients — tell them when they can try again. And always have different tiers: authenticated users get more headroom than anonymous ones.

• Algorithms: token bucket (bursty), sliding window (precise), fixed window (simple)
• Enforce at edge: API gateway/proxy for cheap rejection before app logic
• Distributed state: Redis with atomic operations for multi-server consistency
• Good UX: 429 + Retry-After header, tiered limits by user type

Alright, so when a database is hitting its limits, I'd work through this like a ladder — don't jump to the hardest solution first.

Step one: optimize what you have. Are there missing indexes? N+1 query patterns? Queries scanning full tables? I've seen teams think they need to shard when really they just needed to add a composite index. Check slow query logs, run EXPLAIN on your heavy queries.

Step two: vertical scaling — throw more RAM, faster disks (NVMe), more CPU at it. Sounds crude but honestly it buys time cheaply. Going from a db.r5.large to db.r5.4xlarge might solve your problem for the next 6 months.

Step three: read replicas. If you're read-heavy (most apps are like 90% reads), spin up replicas and route reads there. Your app needs to handle slight replication lag, but for most use cases that's fine.

Step four: caching layer. Put Redis in front for hot data. Cache query results, cache computed aggregations. This offloads the database dramatically.

Step five: if you've done all that and you're STILL hitting limits, now we talk sharding. Horizontal partitioning by some key — user ID, tenant ID, geography. But this is complex — cross-shard queries become hard, transactions across shards are painful. It's a last resort, not a first move.

• Optimize first: indexes, query patterns, connection pooling
• Vertical scale: bigger instance buys time cheaply
• Read replicas: offload read traffic, handle replication lag
• Caching: Redis for hot data, reduce database load by 80%+
• Sharding: last resort — complex but necessary at true scale

The way I think about it — a stateless service is one where any instance can handle any request. No local memory that matters, no "this user's session is on server 3." That's the goal because it makes horizontal scaling trivial.

So basically, you externalize ALL state. Sessions go to Redis or a session store, not in-memory on the app server. File uploads go to S3, not the local filesystem. Caches go to a shared cache layer. Any data that needs to survive a request goes to an external store.

For authentication, use stateless tokens like JWTs instead of server-side sessions. The token carries the claims, any server can validate it without looking anything up. Though honestly, you still want a way to revoke tokens, so maybe a small Redis blacklist for that.

Configuration should come from environment variables or a config service, not local files that differ per instance. And if you need in-memory caching for performance, make sure it's okay for each instance to have its own copy — warm-up on start, invalidate via pub/sub when things change.

The test I use: can I kill any instance at any time and the system keeps working? If yes, you're stateless. If no, find what's stuck on that instance and externalize it.

• Externalize state: sessions → Redis, files → object storage, cache → shared layer
• Stateless auth: JWTs or tokens that don't require server-side lookup
• Config from env: no local files that differ per instance
• Kill test: any instance can die without data loss or user impact

Honestly, this comes down to measurement, not guessing. I've seen teams scale the wrong thing because they assumed the bottleneck without data. So step one — instrument everything and look at the actual numbers.

I'd look at the request flow end-to-end with distributed tracing. Where is time actually spent? If 80% of your p95 latency is in database queries, scaling your app servers does nothing. If your app servers are at 90% CPU but your database is chilling at 20%, you need more compute, not more database.

The metrics I'd focus on: CPU and memory utilization per tier, request queue depth (are requests waiting?), database connection pool exhaustion, error rates by service, and saturation — how close each component is to its theoretical max.

Then I'd apply the theory of constraints — find the bottleneck, fix it, then the bottleneck moves somewhere else. Scale that. Repeat. You're always chasing the weakest link in the chain.

And practically? I'd prioritize by cost-effectiveness. Adding a cache might cost $50/month and 10x your read throughput. Adding 5 more app servers costs $500/month and only 2x's your capacity. Do the cheap, high-impact thing first.

• Measure don't guess: distributed tracing, metrics per tier
• Key signals: CPU, memory, queue depth, connection pool, error rates
• Theory of constraints: find bottleneck → fix → find next bottleneck
• Cost-effectiveness: cache ($50) might beat more servers ($500) for read scaling