🏗️ Cluster-Level Logging Pipelines

Once logs leave the node, they flow into a log backend. Two dominant choices in the Kubernetes ecosystem:

The PLG Stack (Promtail / Loki / Grafana)

Loki is the recommended stack for most greenfield Kubernetes deployments — operationally lightweight and tightly integrated with the Grafana observability suite:

LogQL — Loki's Query Language

# All error logs from the payments namespace
{namespace="payments"} |= "error"

# JSON log parsing + field filter
{namespace="payments", app="checkout"} | json | level="error"

# Rate of error logs per minute (metric query)
rate({namespace="payments"} |= "error" [1m])

# Log lines with a specific trace ID
{namespace="payments"} | json | traceID="abc123xyz"

# Exclude health check noise
{app="my-api"} != "/healthz" != "/readyz"

Structured Logging — the Right Way to Log

Logs as plain text strings are hard to query. Structure them as JSON so the log pipeline can index fields:

// Bad — hard to query, can't filter on level or requestId
log.Printf("ERROR processing request %s: %v", requestID, err)

// Good — structured JSON, Fluent Bit promotes these to labels
log.Info("request failed",
  "level",     "error",
  "requestId", requestID,
  "error",     err.Error(),
  "duration",  elapsed.Milliseconds(),
  "path",      r.URL.Path,
)

Log Retention and Rotation Best Practices

🖥️ Pattern 1 — Node-Level Agent (DaemonSet)

The most operationally efficient pattern. A single Fluent Bit or Fluentd pod per node tails all container log files and forwards them to a backend. No app changes needed.

Fluent Bit — lightweight, preferred for K8s

Fluent Bit is written in C, uses ~1 MB of memory per node, and has native Kubernetes metadata enrichment. It's the default in most managed K8s offerings (GKE, EKS, AKS).

# Install Fluent Bit via Helm (outputs to Loki)
helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit \
  --namespace logging --create-namespace \
  --set config.outputs="[OUTPUT]
    Name loki
    Match *
    Host loki.logging.svc.cluster.local
    Port 3100
    Labels job=fluentbit, namespace=\$kubernetes['namespace_name']"

Fluent Bit config anatomy

# fluent-bit.conf key sections:

[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  parsers.conf

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    multiline.parser  docker, cri   # handle multi-line stack traces
    Tag               kube.*
    Refresh_Interval  5

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Merge_Log           On   # parse JSON logs into structured fields
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[OUTPUT]
    Name  loki
    Match *
    Host  loki.logging.svc.cluster.local
    Port  3100

🔀 Pattern 2 — Sidecar Container

When an application writes logs to a file (not stdout) or needs custom per-app log processing, a sidecar is the right pattern. The sidecar shares a volume with the app container and ships or re-emits the logs.

Two sidecar variants

# Streaming sidecar — re-emit file log to stdout
spec:
  volumes:
  - name: app-logs
    emptyDir: {}
  containers:
  - name: app
    image: myapp:v1
    volumeMounts:
    - name: app-logs
      mountPath: /var/log/app
  - name: log-streamer
    image: busybox:1.36
    args: [/bin/sh, -c, "tail -n+1 -F /var/log/app/app.log"]
    volumeMounts:
    - name: app-logs
      mountPath: /var/log/app

🧠 Knowledge Check