🔭 The Three Pillars of Observability

Production Kubernetes observability requires all three signal types. Each answers a different question during an incident. The Grafana LGTM stack (Loki + Grafana + Tempo + Mimir/Prometheus) provides all three in a cohesive, correlated system.

Install the kube-prometheus-stack (one Helm chart)

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.adminPassword=StrongPass123 \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi

# Installs: Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics
# + pre-built dashboards and alerting rules for all Kubernetes components

📊 Prometheus — Metrics & Alerting

Essential Kubernetes PromQL Queries

# CPU utilisation per pod (%)
100 * sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod, namespace)
  / sum(kube_pod_container_resource_requests{resource="cpu"}) by (pod, namespace)

# Memory utilisation per namespace
sum(container_memory_working_set_bytes{container!=""}) by (namespace)

# Pod restart rate (restarts/hour)
increase(kube_pod_container_status_restarts_total[1h])

# Error rate for a service (5xx / total)
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  / sum(rate(http_requests_total[5m])) by (service)

# P99 request latency
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

# Nodes at >80% CPU
100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (node)) > 80

# PVCs using >80% capacity
100 * (1 - kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes) > 80

PrometheusRule — Alerting Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack   # must match Prometheus selector
spec:
  groups:
    - name: pod-health
      interval: 30s
      rules:
        - alert: PodCrashLooping
          expr: |
            increase(kube_pod_container_status_restarts_total[15m]) > 3
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.pod }} is crash-looping"
            description: "{{ $labels.container }} in {{ $labels.namespace }} restarted {{ $value }} times in 15m"

        - alert: PodNotReady
          expr: |
            kube_pod_status_ready{condition="true"} == 0
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.pod }} not ready for 10 minutes"

        - alert: NodeMemoryPressure
          expr: |
            kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "Node {{ $labels.node }} under memory pressure"

Recording Rules (Pre-computed Aggregates)

# Recording rules speed up dashboards by pre-computing expensive queries
spec:
  groups:
    - name: aggregations
      interval: 1m
      rules:
        - record: namespace:container_cpu_usage_seconds:rate5m
          expr: |
            sum(rate(container_cpu_usage_seconds_total{container!=""}[5m]))
              by (namespace)

        - record: job:http_requests:rate5m
          expr: |
            sum(rate(http_requests_total[5m])) by (job, status)

Alertmanager Routing

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: team-alpha-alerts
  namespace: team-alpha
spec:
  route:
    groupBy: [alertname, namespace]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
    receiver: slack-team-alpha
    routes:
      - matchers:
          - name: severity
            value: critical
        receiver: pagerduty-oncall
  receivers:
    - name: slack-team-alpha
      slackConfigs:
        - apiURL:
            name: slack-webhook-secret
            key: url
          channel: "#alerts-team-alpha"
          title: "{{ .GroupLabels.alertname }}"
    - name: pagerduty-oncall
      pagerdutyConfigs:
        - routingKey:
            name: pd-secret
            key: routingKey

📊 Grafana Dashboards & Loki Logging

Essential Grafana Dashboards

Install Loki (Log Aggregation)

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Loki in simple scalable mode (for production)
helm install loki grafana/loki \
  --namespace monitoring \
  --set loki.storage.type=s3 \
  --set loki.storage.s3.bucketNames.chunks=my-loki-chunks \
  --set loki.storage.s3.region=us-east-1 \
  --set loki.auth_enabled=false

# Grafana Alloy (log shipper) — collects container logs from nodes
helm install alloy grafana/alloy \
  --namespace monitoring \
  --set alloy.configMap.content='
river:
  loki.source.kubernetes "pods" {
    targets    = discovery.kubernetes.pods.targets
    forward_to = [loki.write.default.receiver]
  }
  loki.write "default" {
    endpoint { url = "http://loki-gateway/loki/api/v1/push" }
  }'

LogQL — Querying Logs in Loki

# Stream all logs from a namespace
{namespace="my-app"}

# Filter for errors
{namespace="my-app"} |= "ERROR"

# Filter by pod label + pattern
{namespace="my-app", pod=~"api-.*"} |= "timeout"

# Parse structured JSON logs and filter by field
{namespace="my-app"} | json | level="error" | status_code >= 500

# Rate of error log lines per minute
rate({namespace="my-app"} |= "ERROR" [1m])

# Top 10 slowest requests (parsed from JSON log field)
{namespace="my-app"} | json | line_format "{{.method}} {{.path}} {{.duration_ms}}ms"
  | unwrap duration_ms | quantile_over_time(0.99, [5m]) by (path)

Correlating Logs with Metrics

In Grafana, add a derived field to Loki that extracts trace IDs from log lines and links them directly to Tempo. Click a trace ID in a log line → jump to the full distributed trace.

# Loki data source config (in Grafana) — add derived field
derivedFields:
  - name: TraceID
    matcherRegex: "traceID=(\\w+)"
    url: "${__value.raw}"
    datasourceUid: tempo-uid   # jump to Tempo trace viewer

🔗 Tempo — Distributed Tracing

Tempo stores traces indexed only by trace ID — no full-text search, keeping storage costs low. Grafana's TraceQL lets you search spans by attributes, duration, and status.

Install Tempo

helm install tempo grafana/tempo-distributed \
  --namespace monitoring \
  --set storage.trace.backend=s3 \
  --set storage.trace.s3.bucket=my-tempo-traces \
  --set storage.trace.s3.region=us-east-1

OpenTelemetry SDK Instrumentation (Go example)

// Auto-instrument HTTP server with OTel SDK
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer() {
    exporter, _ := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint("otel-collector.monitoring:4318"),
    )
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithSampler(trace.TraceIDRatioBased(0.1)), // sample 10%
    )
    otel.SetTracerProvider(tp)
}

TraceQL — Querying Traces

# Find slow traces (duration > 2s) for a specific service
{ resource.service.name = "api-gateway" && duration > 2s }

# Find error spans
{ status = error }

# Traces touching both frontend and database
{ resource.service.name = "frontend" } >> { resource.service.name = "postgres" }

# P99 latency across service boundaries
| rate() | quantile(0.99)

Production Observability Tips

📝 Knowledge Check