Chapter 12 Interview: Observability

You need the three pillars working together — metrics, logs, and traces. Any one alone gives you a partial picture. Together they let you detect, diagnose, and resolve issues.

Metrics tell you something is wrong — request rate dropped, error rate spiked, latency is climbing. These are your alerts. You'd use something like Prometheus scraping each service for RED metrics (Rate, Errors, Duration) and USE metrics for infrastructure (Utilization, Saturation, Errors). Dashboards in Grafana give you the at-a-glance view.

Logs tell you what went wrong — structured JSON logs with consistent fields (timestamp, service, request ID, level). Ship them to a centralized system (ELK, Loki) so you can search across all services. Structured logging is non-negotiable — parsing unstructured text at scale is misery.

Traces tell you where it went wrong — a single request's journey across services with timing for each hop. Distributed tracing (Jaeger, Zipkin, or OpenTelemetry) propagates a trace ID through every service so you can see the full waterfall. This is how you find that one slow database query three services deep.

The key insight: metrics for alerting, traces for diagnosis, logs for the details. Connect them — click from a metric spike to the relevant traces to the specific log lines.

• Metrics: RED (Rate/Errors/Duration) for services, USE for infra — alerting
• Logs: structured, centralized, searchable — detailed context
• Traces: end-to-end request flow with timing — localize the problem
• Correlation: link all three via request/trace IDs for fast diagnosis

For payments, you're dealing with people's money — the bar is high. But "five nines" on everything is unrealistic and expensive. You need to be specific about what matters.

Availability SLO: 99.95% — that's about 22 minutes of downtime per month. For a payment service that directly impacts revenue, this is the minimum I'd accept. Measured as: successful responses (non-5xx) divided by total requests over a 30-day rolling window.

Latency SLO: p50 under 200ms, p99 under 1 second. Payments that take more than a second feel broken to users and increase cart abandonment. The p99 matters more than p50 here — tail latency is what creates support tickets.

Correctness SLO: this is the one people forget. 100% of transactions must be accurate — you cannot have phantom charges or lost payments. This means reconciliation checks, idempotency guarantees. Even 99.99% correctness means someone gets wrongly charged every 10,000 transactions. Unacceptable.

I'd define an error budget from these SLOs. If we have 0.05% budget for unavailability and we've burned 80% of it in week two, we freeze deployments and focus on reliability. That's how SLOs become operationally useful — they're a decision-making framework, not just a dashboard number.

• Availability: 99.95% success rate (non-5xx) over 30-day window
• Latency: p50 < 200ms, p99 < 1s — tail latency matters most
• Correctness: 100% transaction accuracy — no room for error
• Error budget: remaining tolerance drives deploy/freeze decisions

This is where distributed tracing earns its keep. Without it you're basically guessing. With it, you have a structured approach.

First, grab the trace ID from the slow request. Every request gets a unique trace ID at the edge (API gateway or first service), and it propagates through every downstream call. Pull up that trace in Jaeger or your tracing tool — you'll see a waterfall view showing every service hop and exactly how long each one took.

Look for the widest span — that's your bottleneck. Is it a database query that took 3 seconds? A downstream service call that timed out and retried? Queue wait time? The trace tells you where the time went. Then zoom into that span's service.

Once you've found the slow service, check its metrics around that timestamp — was CPU pegged? Was it garbage collecting? Connection pool exhausted? Then hit the logs filtered by that trace ID and time window for the specific error or slow operation.

Common culprits I look for: N+1 queries (the trace shows 50 tiny DB calls instead of one), missing indexes (one span shows a 2-second query), synchronous chains that should be parallel (three 500ms calls in sequence = 1.5s that could be 500ms), or a downstream service under load that's responding slowly for everyone.

• Start with trace: find the trace ID, pull up the waterfall
• Find the bottleneck: widest span = where time was spent
• Correlate: metrics + logs for that service at that timestamp
• Common causes: N+1 queries, missing indexes, serial calls, downstream pressure

A circuit breaker is like an electrical one — it trips open to stop cascading damage. When a downstream service starts failing, instead of hammering it with more requests (making things worse), the circuit breaker opens and fails fast locally.

Three states: Closed (normal — requests pass through, failures are counted), Open (tripped — requests immediately fail without calling the downstream, return a fallback or error), and Half-Open (after a timeout, let a few test requests through to see if the service recovered. If they succeed, close the circuit. If they fail, stay open).

You'd use one whenever your service depends on something that can become slow or unavailable — another microservice, a database, a third-party API. Without a circuit breaker, a slow dependency makes you slow, which makes your callers slow — cascading failure across the whole system. With a circuit breaker, you fail fast in 5ms instead of waiting 30 seconds for a timeout.

The key tuning parameters: failure threshold (how many failures before opening — maybe 50% of requests in a 10-second window), recovery timeout (how long to stay open before trying half-open — maybe 30 seconds), and what your fallback is (cached data? degraded response? graceful error?).

• States: Closed → Open (on failures) → Half-Open (test recovery) → Closed
• Purpose: prevent cascading failures, fail fast instead of waiting
• When: any remote dependency — services, databases, third-party APIs
• Tuning: failure threshold, recovery timeout, fallback strategy

Chaos engineering is about proactively finding weaknesses before they find you in production at 3am. But you don't start by killing production servers on day one — that's just chaos, not engineering.

Start small and safe. Game days in staging — pick a failure mode (what if Redis goes down?), form a hypothesis (our app should gracefully degrade to DB queries), run the experiment, observe. Did the system behave as expected? If not, fix it, then re-run. Build confidence incrementally.

Once the team is comfortable, move to production — but with guard rails. Start with blast radius control: affect 1% of traffic, have a kill switch, run during business hours with the team watching. Netflix's Chaos Monkey kills individual instances — that's a great starting point because single-instance failure should always be survivable.

Progressively escalate: instance failure → network partition → dependency failure → AZ failure → region failure. Each level requires more maturity and better tooling. Tools like Gremlin, LitmusChaos, or AWS Fault Injection Simulator give you controlled experiments.

The cultural piece is critical. Chaos experiments are blame-free learning opportunities. When something breaks unexpectedly, that's a success — you found a real vulnerability before customers did. Document findings, fix the gaps, add monitoring for that failure mode, then repeat.

• Start safe: game days in staging with clear hypotheses
• Graduate to prod: small blast radius, kill switches, business hours
• Escalate gradually: instance → network → dependency → zone → region
• Culture: blame-free, findings are wins, fix and repeat