🗄️ Trace Backends — Jaeger vs Grafana Tempo

Installing Grafana Tempo (recommended for new setups)

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Minimal Tempo with local storage (dev/staging)
helm install tempo grafana/tempo \
  --namespace monitoring \
  --set tempo.storage.trace.backend=local

# Production: object storage backend
helm install tempo grafana/tempo-distributed \
  --namespace monitoring \
  --set storage.trace.backend=s3 \
  --set storage.trace.s3.bucket=my-traces-bucket \
  --set storage.trace.s3.region=us-east-1

Installing Jaeger

# Via Jaeger Operator (recommended)
kubectl create namespace observability
kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/latest/download/jaeger-operator.yaml -n observability

# Deploy a Jaeger instance (all-in-one for dev)
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: observability
spec:
  strategy: allInOne    # production: use "production" with Elasticsearch

TraceQL — Query Traces Like Data

Grafana Tempo's TraceQL lets you query spans by attributes, duration, and status — like SQL for traces:

// Find all spans from checkout-svc that took > 500ms
{ .service.name = "checkout-svc" && duration > 500ms }

// Find error spans in the payments service
{ .service.name = "payments-svc" && status = error }

// Find traces that touched both checkout AND payments
{ .service.name = "checkout-svc" } >> { .service.name = "payments-svc" }

// Find DB spans slower than 100ms with specific query
{ span.db.system = "postgresql" && duration > 100ms }

🔗 Connecting the Three Pillars — Exemplars

The real power of observability comes from linking metrics → traces → logs. Exemplars are data points embedded in Prometheus metrics that carry a TraceID — clicking a high-latency data point on a Grafana graph jumps you directly to the trace.

// Expose exemplars from your Go app
histogram.With(prometheus.Labels{"handler": "/checkout"}).
    (Observe)(duration.Seconds(),
        prometheus.Labels{"traceID": span.SpanContext().TraceID().String()})

# In Grafana: enable "Exemplars" on your histogram panel
# → each point shows a TraceID → click → Tempo trace view
# → trace shows every span → click span → Loki logs for that span

Sampling Strategy

🔭 OpenTelemetry — The Unified Standard

OpenTelemetry (OTel, CNCF graduated) is the industry standard for instrumentation. It merges the old OpenTracing and OpenCensus projects into a single API, SDK, and protocol (OTLP). Instrument once, send to any backend.

Instrumenting a Go service

// main.go — set up OTel tracing at startup
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer(ctx context.Context) (*trace.TracerProvider, error) {
    exporter, _ := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector.monitoring:4317"),
        otlptracegrpc.WithInsecure(),
    )
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithSampler(trace.TraceIDRatioBased(0.1)), // 10% sampling
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("checkout-svc"),
            semconv.ServiceVersion("v2.1.0"),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

// Creating spans in handler code
func (s *Server) ProcessOrder(ctx context.Context, req *pb.OrderRequest) (*pb.OrderResponse, error) {
    ctx, span := otel.Tracer("checkout").Start(ctx, "ProcessOrder")
    defer span.End()

    span.SetAttributes(
        attribute.String("order.id",      req.OrderId),
        attribute.String("customer.tier", req.CustomerTier),
        attribute.Int("item.count",       len(req.Items)),
    )

    result, err := s.inventory.CheckStock(ctx, req.Items) // context carries TraceID
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return nil, err
    }
    return result, nil
}

Auto-instrumentation — zero code changes

The OpenTelemetry Operator can inject instrumentation automatically into pods via an annotation — no code changes required for supported languages (Java, Node.js, Python, .NET, Go via eBPF):

# Install the OTel Operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

# Create an Instrumentation CR for Java auto-instrumentation
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-instrumentation
  namespace: production
spec:
  exporter:
    endpoint: http://otel-collector.monitoring:4318
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest

# Annotate the pod/deployment to enable injection
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"

OTel Collector — the routing hub

# otel-collector-config.yaml — receive OTLP, export to Tempo + Jaeger
receivers:
  otlp:
    protocols:
      grpc: { endpoint: "0.0.0.0:4317" }
      http: { endpoint: "0.0.0.0:4318" }

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 512

exporters:
  otlp/tempo:
    endpoint: tempo.monitoring:4317
    tls: { insecure: true }
  jaeger:
    endpoint: jaeger-collector.monitoring:14250
    tls: { insecure: true }
  logging:
    loglevel: warn

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlp/tempo, logging]

Context Propagation — the glue

For traces to span multiple services, the TraceID must be passed in every outbound call. OTel uses the W3C Trace Context standard (traceparent header) by default:

# HTTP header injected automatically by OTel SDK
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^ version  ^^ TraceID (128-bit)               ^^ SpanID  ^^ flags

# gRPC metadata equivalent
grpc-trace-bin: <binary encoded trace context>

# Older B3 format (Zipkin/Jaeger legacy)
X-B3-TraceId: 4bf92f3577b34da6a3ce929d0e0e4736
X-B3-SpanId:  00f067aa0ba902b7
X-B3-Sampled: 1

🧠 Knowledge Check