Distributed Tracing

📘 Chapter 12: Observability & Reliability ⏱️ 9 min read 🏗️ Lesson 051

A user clicks "Place Order" and gets a timeout after 5 seconds. The request touched 10 services. Which one is slow? Where did it fail? Logs tell you something went wrong in each service independently. Metrics tell you which service has high latency. But only distributed tracing shows you the complete journey of that specific request — every service, every database call, every queue message — as a single connected story.

What Is a Trace?

A trace represents the entire journey of a single request through your system. It's composed of spans — each span is one unit of work in one service.

  • Trace: A tree/DAG of spans sharing a single trace ID
  • Span: One operation — has service name, operation name, start time, duration, status, tags
  • Parent-child relationship: A span can spawn child spans (service A calls service B)
  • Root span: The first span — typically the API gateway or edge service
Trace Waterfall — Order Placement Request Service 0ms 500ms 1000ms 1500ms api-gateway 1450ms order-service 1320ms auth-service 35ms inventory-service 85ms payment-service ⚠️ 1050ms (SLOW!) └─ postgres query 980ms — full table scan! 🔍 Root cause: payment-service's DB query does a full table scan (missing index)
Figure 1: A trace waterfall shows exactly where time is spent. The payment-service DB query is the bottleneck — impossible to see from metrics or logs alone.

Trace Context Propagation

How Trace IDs Flow Across Service Boundaries

When service A calls service B, it passes the trace context in request headers:

// W3C Trace Context standard header
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ──  ────────────────────────────────  ────────────────  ──
           version       trace-id                    parent-span-id  flags

Propagation happens via:

  • HTTP headers: traceparent, tracestate (W3C standard)
  • gRPC metadata: Same trace context in gRPC headers
  • Message queues: Trace context in message attributes/headers
  • Databases: SQL comments with trace context (for slow query correlation)

Each service reads the incoming context, creates a child span, and passes the updated context to any downstream calls.

Span Data

What's in a Span?

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "parent_span_id": "a3ce929d0e0e4736",
  "service": "payment-service",
  "operation": "POST /charge",
  "start_time": "2024-01-15T10:23:45.123Z",
  "duration_ms": 1050,
  "status": "error",
  "tags": {
    "http.method": "POST",
    "http.status_code": 504,
    "user.id": "user-12345",
    "payment.amount": 49.99
  },
  "logs": [
    {"time": "...", "message": "Connecting to postgres..."},
    {"time": "...", "message": "Query timeout after 1000ms"}
  ]
}

Sampling Strategies

Tracing every request is expensive at scale (storage, network, CPU). Sampling reduces this:

  • Head-based sampling: Decide at the start whether to trace this request (e.g., trace 1% randomly). Simple but you might miss rare errors.
  • Tail-based sampling: Collect all spans, then decide after seeing the outcome — keep traces that are slow, errored, or interesting. More expensive but catches the important ones.
  • Adaptive sampling: Increase sampling rate when error rates spike; reduce during quiet periods.

Rule of thumb: Always trace 100% of errors and slow requests. Sample normal requests at 1-10%.

OpenTelemetry

OpenTelemetry (OTel) is the CNCF project that provides a single, vendor-neutral standard for instrumentation. Instrument once, send to any backend.

OpenTelemetry Architecture Your Application OTel SDK Auto-instrumentation + Manual spans Metrics + Logs + Traces OTLP OTel Collector Receives → Processes → Exports Filter Sample Batch, enrich, route Vendor-neutral pipeline Jaeger / Tempo (Traces) Prometheus (Metrics) Loki (Logs) or Datadog, Honeycomb, etc. Instrument once with OpenTelemetry → switch backends without code changes
Figure 2: OpenTelemetry provides vendor-neutral instrumentation. The Collector processes and routes telemetry to any backend.

Performance Overhead

Is Tracing Worth the Cost?

  • CPU overhead: ~1-3% for auto-instrumentation (context propagation, span creation)
  • Memory: Buffering spans before export uses some memory (configurable)
  • Network: Exporting span data to collector — batch to reduce overhead
  • Storage: Traces are big (each request generates many spans) — sampling is essential at scale

When it's worth it: Microservices (3+ services), debugging latency issues, understanding dependencies, on-call incident response.

When it's overkill: Simple monoliths, batch jobs, services with trivial call patterns.

Real-World Examples

🏢 Uber & Jaeger

Uber built Jaeger (now a CNCF project) to trace ride requests across 4,000+ microservices. When a rider reports "my app is slow," engineers can pull up the exact trace for that request and see that the delay was in the pricing-service calling a geo-lookup service that had a cold cache. Without tracing across 4,000 services, finding this would take hours of log correlation. With Jaeger, it takes seconds.

🏢 Slow API Debugging

A team notices their /api/orders endpoint has p99 latency of 2 seconds (SLO: 500ms). Metrics show the order-service itself is fast (50ms). A trace reveals the truth:

  • order-service → inventory-service: 50ms ✅
  • order-service → payment-service: 80ms ✅
  • order-service → notification-service: 1800ms ❌
  • notification-service → email-provider-api: 1750ms (third-party timeout!)

Fix: Make the notification call asynchronous (fire-and-forget via message queue). Latency drops to 150ms. Without the trace, the team would have optimized the wrong service.

Interactive: Build a Trace

🔗 Trace a Request Through Services

Click services in order to build a trace. Each click adds a span to the waterfall.