Containers are ephemeral. When one exits, its filesystem — including any log files — disappears. Observability means capturing logs, metrics, and traces outside the container, before it's gone.

1. The Observability Challenge

Traditional servers are long-lived. You SSH in, tail a log file, and the data is always there. Containers throw that model away: a container can be replaced in seconds, scaled to dozens of replicas, or killed by a health-check failure. Any data written inside the container is gone with it.

This forces a discipline shift. Instead of checking one server, you need:

• Centralised log aggregation — all container output flows to one place
• Time-series metrics — numeric snapshots scraped from every replica
• Distributed traces — a request fingerprint that follows work across many containers

2. Logging: stdout/stderr Is the Contract

The Twelve-Factor App (Factor XI) is unambiguous: write logs to stdout. Docker captures every byte written to stdout and stderr, stores it via the configured log driver, and makes it accessible with docker logs. Writing to a file inside the container breaks this chain.

# ✅ View live logs from a container
docker logs -f my-api

# ✅ Last 100 lines with timestamps
docker logs --tail 100 --timestamps my-api

# ❌ Don't do this inside your container
echo "error" >> /var/log/app/error.log   # lost on container removal

3. Log Drivers

Docker's log driver controls where stdout/stderr goes after Docker captures it. The default is json-file — fine for development, not for production at scale.

Configure the driver per container in Compose or globally in /etc/docker/daemon.json:

# docker-compose.yml — per service
services:
  api:
    image: my-api:latest
    logging:
      driver: "json-file"
      options:
        max-size: "10m"    # rotate at 10 MB
        max-file: "5"      # keep 5 rotated files

# /etc/docker/daemon.json — global default
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

4. Structured Logging

Plain text logs are hard to query at scale. Structured logging means emitting JSON objects so every log line is machine-parseable. An aggregator can then index individual fields without regex.

# Unstructured — hard to query
2024-01-15T10:23:01Z ERROR payment service: charge failed for user 4892 amount=99.99

# Structured JSON — indexable, filterable
{
  "timestamp": "2024-01-15T10:23:01Z",
  "level":     "error",
  "service":   "payment",
  "event":     "charge_failed",
  "user_id":   4892,
  "amount":    99.99,
  "trace_id":  "abc123def456"
}

Key fields every log line should carry:

• timestamp — ISO 8601, always UTC
• level — debug / info / warn / error
• service — which container / microservice emitted it
• trace_id — links logs to the distributed trace for that request
• message — human-readable description

The EFK stack (Elasticsearch + Fluent Bit + Kibana) is a popular open-source choice: Fluent Bit ships logs from each container to Elasticsearch, and Kibana provides a search and visualisation UI.

5. Container Metrics

Metrics answer "how is the container behaving right now?" Docker exposes real-time stats via the kernel's cgroups interface, which limits and tracks resource usage per container.

# Live stats for all running containers
docker stats

# One-shot snapshot (no streaming)
docker stats --no-stream

# Output
CONTAINER ID   NAME       CPU %   MEM USAGE / LIMIT    NET I/O        BLOCK I/O
a1b2c3d4e5f6   api        1.23%   128MiB / 512MiB      10MB / 2MB     0B / 4MB

Key metrics to watch in production:

• CPU % — sustained >80% → scale out or profile
• Memory usage / limit — approaching the limit triggers OOM kills
• Network I/O — unexpected spikes may indicate a loop or attack
• Block I/O — high write volume may mean you need a volume, not the container layer
• Restart count — a container that keeps restarting is a red flag

cAdvisor (Container Advisor) is a Google-maintained daemon that scrapes cgroup metrics from every container on a host and exposes them in Prometheus format — the standard bridge between Docker and the Prometheus ecosystem.

6. Prometheus & Grafana Pattern

The most common open-source metrics stack for containers is Prometheus + Grafana:

1. Your app (or cAdvisor) exposes a /metrics HTTP endpoint in Prometheus text format
2. Prometheus scrapes that endpoint on a configured interval (e.g. every 15 s)
3. Grafana queries Prometheus and renders dashboards, graphs, and alerts

# prometheus.yml — minimal scrape config
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]

  - job_name: "my-api"
    static_configs:
      - targets: ["api:9090"]  # app exposes /metrics here

7. Distributed Tracing

In a microservices architecture, a single user request may pass through a gateway, three services, a cache, and a database — each in a different container. Distributed tracing links all that work into a single timeline called a trace, made of spans.

OpenTelemetry (OTel) is the CNCF-standard SDK for instrumenting apps. It produces traces (and metrics, and logs) in a vendor-neutral format that backends like Jaeger and Zipkin can store and display.

# docker-compose.yml — Jaeger all-in-one for dev tracing
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"   # Jaeger UI
      - "4317:4317"     # OTLP gRPC receiver
    environment:
      - COLLECTOR_OTLP_ENABLED=true

  api:
    image: my-api:latest
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317
      - OTEL_SERVICE_NAME=my-api

The key mechanism is context propagation: when service A calls service B, it injects a traceparent HTTP header. Service B reads it and creates a child span under the same trace ID. The result is a waterfall view in the Jaeger UI showing exactly where latency lives.

8. The Three Pillars

Logs, metrics, and traces are complementary, not redundant. Use them together:

Hands-On Task: Prometheus + Grafana + Sample App

# compose.yml
services:
  app:
    image: nginx:alpine          # sample workload
    ports: ["8080:80"]

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    privileged: true
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports: ["8081:8080"]

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports: ["9090:9090"]

  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

global:
  scrape_interval: 10s
scrape_configs:
  - job_name: cadvisor
    static_configs:
      - targets: ["cadvisor:8080"]

docker compose up -d

Interactive Quizzes

Key Takeaways

• Containers are ephemeral — capture logs and metrics externally or lose them forever
• stdout is the contract — write all log output to stdout; Docker log drivers handle the rest
• Log drivers (json-file, fluentd, awslogs, gcplogs) route output to the right sink; always configure rotation in production
• Structured (JSON) logs are indexable — include timestamp, level, service, and trace_id on every line
• docker stats gives instant metrics; cAdvisor exposes them in Prometheus format for persistent collection
• Prometheus + Grafana is the dominant open-source pattern: scrape /metrics, store time-series, visualise dashboards
• Distributed tracing (OpenTelemetry + Jaeger/Zipkin) links work across containers via a shared trace ID — essential for debugging microservices latency
• Three pillars: Logs (what), Metrics (how much), Traces (where) — use all three; they answer different questions
• Managed platforms (Datadog, New Relic, Grafana Cloud) unify all three pillars — worth the cost for teams that want to skip operating the stack themselves