🎯 ServiceMonitor — Telling Prometheus What to Scrape

The Prometheus Operator introduces CRDs that let you configure scraping declaratively in Kubernetes. A ServiceMonitor selects Services by label and tells Prometheus which port and path to scrape. No more editing prometheus.yml by hand.

# Your app Service (must have a named port)
apiVersion: v1
kind: Service
metadata:
  name: my-app
  namespace: production
  labels:
    app: my-app
    monitoring: "true"   # ← ServiceMonitor selects on this
spec:
  ports:
  - name: metrics        # named port — required by ServiceMonitor
    port: 9090
  selector:
    app: my-app

---
# ServiceMonitor — discovers and scrapes the Service
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: monitoring   # can live in a different namespace
spec:
  namespaceSelector:
    matchNames: [production]
  selector:
    matchLabels:
      monitoring: "true"
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
    scheme: http

🚨 PrometheusRule — Alerts as Code

A PrometheusRule defines recording rules and alert rules as Kubernetes resources. The Prometheus Operator watches for them and automatically reloads Prometheus configuration.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-alerts
  namespace: monitoring
  labels:
    prometheus: kube-prometheus   # must match Prometheus CR's ruleSelector
    role: alert-rules
spec:
  groups:
  - name: my-app.rules
    rules:

    # Recording rule — pre-compute expensive query
    - record: job:http_requests:rate5m
      expr: rate(http_requests_total[5m])

    # Alert — pod restart storm
    - alert: PodRestartingTooFast
      expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} restarting frequently"
        description: "{{ $labels.pod }} in {{ $labels.namespace }} restarted {{ $value }} times in 15m"
        runbook_url: "https://wiki.example.com/runbooks/pod-restarts"

    # Alert — high error rate
    - alert: HighErrorRate
      expr: >
        rate(http_requests_total{status=~"5.."}[5m])
          / rate(http_requests_total[5m]) > 0.05
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Error rate above 5% for {{ $labels.service }}"

Alert lifecycle — pending → firing

The for field prevents alert flapping: the condition must be true for that duration before the alert transitions from pending to firing. Alertmanager then routes, deduplicates, and silences.

📈 Grafana Dashboards

Grafana is the visualisation layer. kube-prometheus-stack ships ~30 pre-built dashboards covering nodes, pods, namespaces, API server, etcd, and more.

Essential built-in dashboards

Dashboard-as-Code with ConfigMap

# Grafana auto-discovers dashboards from ConfigMaps with the right label
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"   # sidecar discovers this label
data:
  my-app.json: |
    { "title": "My App", "panels": [...] }   # paste exported Grafana JSON

🔥 Prometheus — The Metrics Standard

Prometheus is the de-facto metrics system for Kubernetes (CNCF graduated). It uses a pull model — Prometheus scrapes HTTP /metrics endpoints on a schedule, stores time-series data locally, and evaluates alert rules.

Installing kube-prometheus-stack (recommended)

The kube-prometheus-stack Helm chart installs Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter, and kube-state-metrics in one shot — with sane defaults for Kubernetes monitoring:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.adminPassword=changeme \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi

Key exporters to know

Essential PromQL patterns

# CPU usage % per pod (across all namespaces)
100 * sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod, namespace)
  / sum(kube_pod_container_resource_requests{resource="cpu"}) by (pod, namespace)

# Memory usage vs limit (% — alert if > 80%)
container_memory_working_set_bytes{container!=""}
  / container_spec_memory_limit_bytes{container!=""} * 100

# Pod restart rate — alert if > 0 in last 5m
increase(kube_pod_container_status_restarts_total[5m]) > 0

# HTTP error rate for a service
rate(http_requests_total{status=~"5.."}[5m])
  / rate(http_requests_total[5m]) * 100

# p99 request latency
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

🧠 Knowledge Check