SLOs, SLIs, and Error Budgets

📘 Chapter 12: Observability & Reliability ⏱️ 8 min read 🏗️ Lesson 050

"We need to be more reliable" is vague. How reliable? Measured how? And what happens when we're not? SLOs, SLIs, and error budgets turn reliability from a feeling into a measurable engineering practice with clear tradeoffs between reliability and feature velocity.

The Hierarchy: SLI → SLO → SLA

SLI → SLO → SLA Pyramid SLA Service Level Agreement Contract with consequences ($$$) SLO Service Level Objective Internal target (stricter than SLA) SLI Service Level Indicator The actual measurement (metrics) e.g., "99.9% uptime or customer gets credit" e.g., "99.95% of requests succeed within 200ms" e.g., "% of requests with latency < 200ms" SLOs are always stricter than SLAs — you want internal alerts before breaking your contract
Figure 1: SLIs are what you measure, SLOs are your targets, SLAs are contractual promises with financial consequences.

SLI — Service Level Indicator

A carefully defined quantitative measure of some aspect of your service. Good SLIs:

  • Availability: % of successful requests (HTTP 2xx/3xx out of total)
  • Latency: % of requests faster than a threshold (p99 < 200ms)
  • Correctness: % of requests returning the right answer
  • Freshness: % of data updated within X seconds

⚠️ Use percentiles for latency, not averages! A p50 of 50ms and p99 of 5000ms means 1% of users wait 100× longer — an average of 100ms hides this completely.

SLO — Service Level Objective

A target value for your SLI over a time window:

  • "99.9% of requests complete successfully in a rolling 30-day window"
  • "99th percentile latency < 200ms over each calendar month"
  • "Data freshness: 99.5% of records updated within 60 seconds"

Choose SLOs based on user expectations, not engineering pride. A batch job doesn't need 99.99%.

SLA — Service Level Agreement

A contract between provider and customer. If you violate it, there are consequences: service credits, refunds, or contract termination. SLAs are always looser than internal SLOs — you want to catch problems before they become contractual violations.

The Math of Nines

Availability Downtime/Year Downtime/Month Typical For
99% ("two nines") 3.65 days 7.3 hours Internal tools
99.9% ("three nines") 8.7 hours 43.8 min Most SaaS products
99.95% 4.4 hours 21.9 min Cloud providers
99.99% ("four nines") 52.6 min 4.4 min Payment systems
99.999% ("five nines") 5.3 min 26 sec 911 systems, pacemakers

Each additional nine is 10× harder and more expensive. Going from 99.9% → 99.99% doesn't sound like much, but it means 10× less allowed downtime.

Error Budgets

Your error budget is the gap between 100% and your SLO. If your SLO is 99.9% availability over 30 days, your error budget is 0.1% = 43.8 minutes of allowed downtime per month.

What Error Budgets Enable

  • Permission to take risks: As long as budget remains, ship features, experiment, deploy often
  • Objective decision-making: "We've used 30% of our budget — we can still do the risky migration"
  • Alignment between teams: Product wants features, SRE wants reliability — error budgets give both a shared framework
  • Clear consequences: When budget is exhausted, freeze features and focus on reliability
Error Budget Burn-Down (30-Day Window) 100% 75% 50% 25% 0% Budget Remaining Day 1 Day 10 Day 20 Day 30 Expected uniform burn Deployment incident (-15% budget) DB failover (-25% budget) ⚠️ Danger zone — freeze deployments, focus on reliability Actual budget remaining End of month: 10% budget left ✅ (healthy)
Figure 2: Error budget burn-down chart. Incidents consume budget; if you hit zero, stop risky changes.

When Error Budget is Exhausted

Budget Policies (agree on these in advance!)

  • Freeze feature deployments — only reliability improvements and bug fixes
  • Conduct post-mortems on incidents that consumed budget
  • Invest in automation — better testing, canary deployments, rollback speed
  • Re-evaluate SLOs — maybe your target is unrealistically tight for your architecture

The freeze isn't punishment — it's the natural consequence of spending your reliability budget. Just like a financial budget.

Choosing Good SLIs

SLI Best Practices

  • Measure from the user's perspective: Client-side latency, not server-side. Include network time.
  • Use percentiles, not averages: p50 (median experience), p99 (worst 1% of users)
  • Separate success from failure latency: A fast error (10ms 500) shouldn't improve your latency SLI
  • Choose indicators users actually care about: Playback start time matters more than API response time for a streaming service
  • Keep it simple: 3-5 SLIs per service maximum. More creates confusion.

Real-World Examples

🏢 Google: Error Budgets in Practice

Google's SRE teams and product teams share ownership of the error budget. If a product team's feature breaks reliability enough to exhaust the budget, their next sprint pivots to reliability work. This creates a natural feedback loop: teams that ship carefully get to ship more often, while reckless teams get slowed down. The result is that Google's services maintain high reliability while still shipping features rapidly.

🏢 Spotify: Differentiated SLOs

Not all features deserve the same SLO. Spotify sets different targets for different user journeys:

  • Music playback: 99.99% availability (core value proposition — must never fail)
  • Search: 99.9% availability (important but brief degradation is tolerable)
  • Recommendations: 99% availability (nice-to-have — fallback to popular tracks)
  • Account settings: 99.5% (rarely used, can be briefly unavailable)

This lets engineering invest reliability budget where it matters most to users.

Interactive: SLO Calculator

📊 SLO & Error Budget Calculator

Set your availability target and see your error budget:

99.9%

Simulate an Incident