Chaos Engineering Principles

📘 Chapter 12: Observability & Reliability ⏱️ 9 min read 🏗️ Lesson 053

You've built circuit breakers, fallbacks, redundancy, and auto-scaling. But do they actually work? The only way to know is to test them under real conditions. Chaos engineering is the discipline of deliberately injecting failures into production systems to build confidence that they can withstand turbulent conditions — finding weaknesses before your users do.

The Core Idea

Chaos engineering is NOT breaking things for fun. It's a rigorous, scientific practice:

  • Define steady state (normal behavior — what metrics look like when things are healthy)
  • Form a hypothesis ("if we kill 1 instance, traffic will shift to healthy instances with no user impact")
  • Inject failure (actually kill the instance in production)
  • Observe (did the system behave as hypothesized?)
  • Learn (if not, you found a weakness — fix it!)

If the hypothesis holds, you've increased confidence. If it doesn't, you've found a bug before it caused an outage.

Chaos Engineering Loop 1. Define Steady State "Error rate < 0.1%, p99 < 200ms" 2. Hypothesize "System will handle this failure" 3. Inject Failure "Kill instance / add latency" 4. Observe & Verify "Did steady state hold?" 5. Improve & Repeat "Fix weakness, expand scope" Each iteration increases confidence in resilience
Figure 1: Chaos engineering is a continuous loop — define, hypothesize, experiment, observe, improve, repeat.

Types of Chaos Experiments

Experiment What It Tests Expected Behavior
Kill instances Auto-scaling, load balancing, redundancy Traffic routes to healthy instances
Inject latency Timeouts, circuit breakers, SLOs Circuit opens, fallback activates
Network partition Split-brain handling, data consistency System detects partition, degrades gracefully
Fill disk Disk pressure alerts, log rotation Alert fires, cleanup runs, no crash
Clock skew Time-dependent logic, cert validation No silent data corruption
DNS failure DNS caching, service discovery Cached records used, graceful retry

Blast Radius Control

Never start big. Expand the blast radius gradually as confidence grows:

Blast Radius Expansion Single Pod Minimal risk Start here! Single Node ~5% of capacity Single AZ ~33% of capacity Entire Region Advanced — Netflix level
Figure 2: Start with the smallest blast radius and expand only after building confidence at each level.

Game Days

Planned Chaos with the Team Ready

A game day is a scheduled chaos experiment where:

  • The team is on standby and aware (not a surprise!)
  • There's a clear hypothesis and success criteria
  • A "red button" to abort if things go worse than expected
  • Post-experiment retrospective to capture learnings
  • Runbooks are updated based on findings

Game days build team muscle memory for incident response — the same way fire drills prepare people for real fires.

Prerequisites

You Need These BEFORE Doing Chaos Engineering

  • Observability: Metrics, logs, and traces to detect impact (Lesson 49)
  • SLOs: A definition of "steady state" to verify against (Lesson 50)
  • Circuit breakers: Protection against cascading failures (Lesson 52)
  • Graceful degradation: Fallbacks that activate when things break
  • Automated rollback: Ability to quickly undo the experiment
  • On-call rotation: Someone ready to respond if it goes wrong

Doing chaos engineering without observability is like doing science without instruments — you can't measure anything.

Tools

Tool Creator Specialty
Chaos Monkey Netflix Randomly kills instances in production
Gremlin Gremlin Inc. Commercial platform — network, resource, state attacks
Litmus CNCF Kubernetes-native chaos experiments
AWS FIS Amazon Fault Injection Simulator for AWS services
Chaos Toolkit Open source Declarative, extensible chaos experiments

Real-World Examples

Netflix: The Simian Army

Netflix's chaos tools run continuously in production:

  • Chaos Monkey: Kills random instances during business hours
  • Latency Monkey: Injects artificial delays into service calls
  • Chaos Kong: Simulates an entire AWS region going offline
  • Conformity Monkey: Shuts down instances that don't follow best practices

The philosophy: if your service can't survive Chaos Monkey killing instances randomly every day, it shouldn't be in production. This forces engineers to build resilient services from day one.

Amazon: Prime Day Game Days

Weeks before Prime Day (their highest-traffic event), Amazon runs game days where they deliberately disable entire subsystems: "What happens if the recommendations engine goes down during peak traffic? What if a database shard fails?" Teams practice their response, verify that fallbacks work, and fix gaps. By the time Prime Day arrives, they've already seen and handled their worst-case scenarios. This is why Prime Day rarely has major outages despite 100x normal traffic.

Interactive: Design a Chaos Experiment

Design Your Chaos Experiment

Choose what to break, predict the outcome, then see what happens.