Monitoring: Metrics, Logs, Traces

📘 Chapter 12: Observability & Reliability ⏱️ 9 min read 🏗️ Lesson 049

When your system is running in production, you can't attach a debugger. You need observability — the ability to understand what's happening inside your system by examining its outputs. The three pillars of observability are metrics, logs, and traces. Each answers different questions, and together they give you a complete picture.

The Three Pillars of Observability

Three Pillars of Observability 📊 Metrics Numbers over time • CPU usage: 78% • Request rate: 1,200/s • Error rate: 0.5% • p99 latency: 230ms Questions answered: "Is something broken?" "What's the trend?" "When did it start?" "Are we within SLO?" Tools: Prometheus, Datadog 📝 Logs Discrete events • Error stack traces • Request details • State transitions • Audit trail Questions answered: "What exactly happened?" "What was the error?" "What was the input?" "Who did what?" Tools: ELK, Loki, Splunk 🔗 Traces Request paths • Service-to-service flow • Latency per hop • Dependency mapping • Bottleneck identification Questions answered: "Where is the time going?" "Which service is slow?" "What's the call graph?" "Where did it fail?" Tools: Jaeger, Zipkin, Tempo
Figure 1: The three pillars each answer different questions — you need all three for full observability.

Metrics: Numbers Over Time

Metric Types

  • Counter: Only goes up. Requests served, errors occurred, bytes sent. You derive rates from counters (requests/second).
  • Gauge: Goes up and down. Current CPU usage, active connections, queue depth.
  • Histogram: Distribution of values. Request latency buckets (how many requests took 0-10ms, 10-50ms, 50-100ms, etc.).

USE Method (for resources)

For every resource (CPU, memory, disk, network), ask:

  • Utilization — What percentage of the resource is busy?
  • Saturation — How much extra work is queued (waiting)?
  • Errors — How many error events occurred?

Best for: infrastructure monitoring (servers, databases, queues)

RED Method (for services)

For every service, measure:

  • Rate — Requests per second
  • Errors — Failed requests per second
  • Duration — Latency distribution (histograms, not just averages!)

Best for: request-driven microservices

Logs: Discrete Events

Structured vs Unstructured Logs

Unstructured (hard to query):

2024-01-15 10:23:45 ERROR Payment failed for user 12345 - timeout connecting to stripe

Structured (JSON) (easy to query, filter, aggregate):

{
  "timestamp": "2024-01-15T10:23:45Z",
  "level": "error",
  "service": "payment-service",
  "message": "Payment failed",
  "user_id": "12345",
  "error": "timeout",
  "dependency": "stripe",
  "trace_id": "abc123def456",
  "duration_ms": 5000
}

Structured logs are searchable: "show me all errors from payment-service where dependency=stripe in the last hour."

Log Levels

  • DEBUG: Verbose details for development (never in prod at scale)
  • INFO: Normal operations (request received, job completed)
  • WARN: Something unexpected but handled (retry succeeded, cache miss)
  • ERROR: Something failed, needs attention (request failed, dependency down)
  • FATAL: Process cannot continue (out of memory, config missing)

Correlation IDs

A unique ID (often the trace ID) attached to every log line for a single request. When a user reports "my payment failed," you search logs by their correlation ID and see every log line across every service for that specific request.

Traces: Request Paths

A trace follows a single request as it flows through multiple services. Each service adds a span — a record of what it did and how long it took. Together, the spans form a tree showing the complete journey.

How They Complement Each Other

Metrics alert you that something is wrong (error rate spiked!).
Logs explain what happened (NullPointerException in payment handler).
Traces show where in the call chain it happened (the 3rd service in the request path).

Incident Investigation Flow 1. METRIC ALERT Error rate jumped from 0.1% → 5% 2. DASHBOARD Narrow to service: payment-service 3. SEARCH LOGS Filter: level=error → "DB timeout" errors 4. TRACE REQUEST DB query taking 8s → missing index! ✅ ROOT CAUSE FOUND Missing database index on orders.user_id Fix: CREATE INDEX idx_orders_user_id ON orders(user_id) Total investigation time: ~5 minutes (with good observability) Without observability: hours of guessing and deploying debug code
Figure 2: Metrics → Logs → Traces — each pillar narrows the search until you find root cause.

The 4 Golden Signals (Google SRE)

Google's Site Reliability Engineering book recommends monitoring these four signals for every service:

  • Latency: Time to serve a request (distinguish successful vs failed requests)
  • Traffic: Demand on your system (requests/sec, sessions, reads/writes)
  • Errors: Rate of failed requests (HTTP 5xx, timeouts, wrong results)
  • Saturation: How "full" your service is (CPU, memory, queue depth)

If you can only monitor four things, monitor these.

Tools Landscape

Pillar Open Source Commercial
Metrics Prometheus + Grafana Datadog, New Relic
Logs ELK Stack, Loki Splunk, Datadog Logs
Traces Jaeger, Zipkin, Tempo Datadog APM, Lightstep
Unified OpenTelemetry (instrumentation standard) Datadog, Honeycomb

OpenTelemetry is the emerging standard — instrument once, send to any backend. It provides APIs and SDKs for metrics, logs, and traces in a vendor-neutral way.

Real-World Examples

🏢 Datadog: Unified Observability

Datadog's power comes from correlating all three pillars. When you see a latency spike on a dashboard (metric), you click to see logs from that time window, then click a trace ID to see the full request waterfall. The correlation is automatic — no manual searching across systems. This is why unified platforms are winning over separate tools.

🏢 Netflix: RED Method at Scale

Netflix monitors hundreds of microservices using the RED method. Every service automatically reports Rate, Errors, and Duration. Their tool Atlas ingests ~2 billion metrics per minute. When a deployment increases p99 latency by 50ms, automated canary analysis (Kayenta) detects it within minutes and rolls back — often before any user notices.

Interactive: Incident Investigation

🔍 Debug the Outage

Alert: Error rate spiked to 12% on checkout-service! Use the three pillars to find root cause.