Circuit Breakers & Graceful Degradation
Your recommendation service goes down. Without protection, every request to your product page now waits 30 seconds for a timeout, backing up threads, exhausting connection pools, and eventually bringing down your entire website — even though the product page could work fine without recommendations. This is a cascading failure, and the circuit breaker pattern prevents it.
The Problem: Cascading Failures
In microservices, Service A depends on Service B which depends on Service C. When C becomes slow (not down, just slow):
- B's threads are all blocked waiting for C → B becomes slow
- A's threads are all blocked waiting for B → A becomes slow
- Users' requests queue up → everything times out → total system failure
The irony: a single slow service causes more damage than a completely dead one (dead services fail fast; slow services consume resources while failing).
Circuit Breaker Pattern
Inspired by electrical circuit breakers: when too much current flows, the breaker trips to prevent a fire. In software, when a dependency fails too often, the circuit breaker opens to prevent cascading failures.
Configuration Parameters
- Failure threshold: How many failures (or what % failure rate) triggers the circuit to open (e.g., 5 failures in 10 seconds, or 50% failure rate)
- Timeout (reset timer): How long to wait in Open state before trying Half-Open (e.g., 30 seconds)
- Half-open retry count: How many test requests to allow in Half-Open (e.g., 3 successes → close)
- Monitoring window: Time window for counting failures (sliding window or count-based)
Fallback Strategies
When the Circuit is Open, Return...
- Cached response: Return the last known good value (stale data is often better than no data)
- Default value: Generic recommendations, empty list, placeholder content
- Degraded feature: Show the page without personalization rather than breaking entirely
- Queue for later: Accept the request but process it when the dependency recovers
- Clear error message: "Recommendations temporarily unavailable" (honest, not a crash)
The key principle: partial functionality is always better than total failure.
Bulkhead Pattern
Named after ship compartments: if one compartment floods, watertight doors prevent the entire ship from sinking. In software, isolate dependencies so one failing dependency can't consume all your resources.
Timeout Budgets
Cascading Timeouts
If your API has a 3-second SLO, and it calls 3 services sequentially, you can't give each service 3 seconds — they'd stack to 9 seconds. Instead, allocate a timeout budget:
- Total budget: 3000ms
- Auth service: 200ms timeout (fast, critical)
- Inventory service: 500ms timeout
- Payment service: 2000ms timeout (complex, needs more time)
- Reserve: 300ms for overhead and retries
Rule: Each downstream timeout must be less than the caller's remaining budget. Otherwise you'll timeout at the edge before getting a response.
Real-World Examples
🏢 Netflix: Hystrix & Resilience4j
Netflix pioneered the circuit breaker pattern with Hystrix (now succeeded by resilience4j). Every one of Netflix's hundreds of microservices wraps external calls in circuit breakers. When a dependency degrades, the circuit opens and a fallback is served. Netflix's principle: "Everything fails, all the time." Their architecture assumes every dependency will fail and pre-plans the degraded experience. The result: even when AWS regions go down, Netflix keeps streaming.
🏢 Amazon.com: Graceful Degradation
Amazon's product page calls 200-300 services. When the recommendations engine fails:
- ❌ Bad: Show an error page (lost sale)
- ✅ Good: Show generic popular items instead of personalized recommendations
When the reviews service fails: show product without reviews. When pricing fails: show "temporarily unavailable" price but let users add to wishlist. Every non-critical feature has a fallback. The product page never fully breaks because of a non-critical dependency.
Interactive: Circuit Breaker Simulation
⚡ Watch a Circuit Breaker in Action
Click "Send Request" to call a flaky downstream service. Watch the circuit breaker respond to failures.