📐 The Capacity Planning Cycle

Capacity planning is not a one-time event — it's a continuous loop that runs alongside your normal operations cadence. The goal is never to be surprised by resource exhaustion, but also never to massively over-provision.

📊 Measuring Utilisation & Setting Headroom

Node Allocatable vs Capacity

Kubernetes reserves resources for the OS and kubelet. The allocatable amount is what pods can actually request — always less than the node's physical capacity.

# View allocatable vs capacity for all nodes
kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,\
CPU-CAP:.status.capacity.cpu,\
CPU-ALLOC:.status.allocatable.cpu,\
MEM-CAP:.status.capacity.memory,\
MEM-ALLOC:.status.allocatable.memory'

# Typical overhead on a 4-CPU / 16Gi node:
# capacity.cpu:    4000m
# allocatable.cpu: 3800m   (200m reserved for system)
# capacity.memory: 16Gi
# allocatable.memory: 15.4Gi  (~600Mi reserved)

Cluster-Wide Utilisation PromQL

# Requested CPU as % of allocatable (across all nodes)
sum(kube_pod_container_resource_requests{resource="cpu"})
  / sum(kube_node_status_allocatable{resource="cpu"}) * 100

# Actual CPU usage as % of allocatable
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m]))
  / sum(kube_node_status_allocatable{resource="cpu"}) * 100

# Memory requested as % of allocatable
sum(kube_pod_container_resource_requests{resource="memory"})
  / sum(kube_node_status_allocatable{resource="memory"}) * 100

# Per-node: how much of its allocatable CPU is requested
sum by (node) (kube_pod_container_resource_requests{resource="cpu"})
  / sum by (node) (kube_node_status_allocatable{resource="cpu"}) * 100

Utilisation Snapshot

Example: CPU has healthy headroom; memory actual usage is dangerously high — add nodes or right-size memory limits.

Headroom Zones

📈 Forecasting Growth & Node Sizing

Simple Linear Forecast with PromQL

# Project CPU request growth 30 days into the future
# based on the last 14 days of trend
predict_linear(
  sum(kube_pod_container_resource_requests{resource="cpu"})[14d:1h],
  30 * 24 * 3600   # 30 days in seconds
)

# Example interpretation:
# Current: 120 CPU cores requested
# predict_linear result: 180 cores in 30 days
# → Need to add ~60 cores (e.g. 4 × m5.4xlarge) within 30 days

# Memory forecast
predict_linear(
  sum(kube_pod_container_resource_requests{resource="memory"})[14d:1h],
  30 * 24 * 3600
)

Node Size Trade-offs

The Three Autoscaling Layers

# Layer them correctly:
# 1. VPA (Off mode) → gives right-sized requests
# 2. HPA → scales replicas on CPU/RPS
# 3. Karpenter → adds nodes when HPA can't fit new pods

# Don't run VPA Auto + HPA on the same workload —
# VPA evicting pods conflicts with HPA scaling targets.
# Use VPA for right-sizing, HPA for throughput scaling.

Capacity Alerts

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: capacity-alerts
  namespace: monitoring
spec:
  groups:
    - name: capacity
      rules:
        # Node CPU requested > 80% of allocatable
        - alert: NodeCPURequestsHigh
          expr: |
            sum by (node) (kube_pod_container_resource_requests{resource="cpu"})
            / sum by (node) (kube_node_status_allocatable{resource="cpu"}) > 0.80
          for: 15m
          labels: { severity: warning }
          annotations:
            summary: "Node {{ $labels.node }} CPU requests at {{ $value | humanizePercentage }}"

        # Cluster memory will be full in < 7 days
        - alert: ClusterMemoryWillExhaustSoon
          expr: |
            predict_linear(
              sum(kube_node_status_allocatable{resource="memory"}
                - on(node) group_left()
                sum by (node)(kube_pod_container_resource_requests{resource="memory"})
              )[3d:1h], 7 * 24 * 3600) < 0
          for: 1h
          labels: { severity: critical }
          annotations:
            summary: "Cluster memory capacity will be exhausted within 7 days"

💵 Chargeback & Resource Governance

Kubecost — Cost Attribution

helm repo add kubecost https://kubecost.github.io/cost-analyzer
helm repo update

helm install kubecost kubecost/cost-analyzer \
  --namespace kubecost \
  --create-namespace \
  --set kubecostToken="" \
  --set global.prometheus.fqdn=http://kube-prometheus-stack-prometheus.monitoring:9090

# Access UI
kubectl port-forward -n kubecost svc/kubecost-cost-analyzer 9090:9090

Key Kubecost Queries (API)

# Cost per namespace last 30 days
curl "http://localhost:9090/model/allocation?window=30d&aggregate=namespace"

# Cost per team label last 7 days
curl "http://localhost:9090/model/allocation?window=7d&aggregate=label:team"

# Identify idle/wasted CPU cost
curl "http://localhost:9090/model/savings/requestSizingV2"
# Returns per-workload recommendations:
# {
#   "namespace": "team-alpha",
#   "workload": "api-server",
#   "currentCPURequest": "2",
#   "recommendedCPURequest": "0.4",
#   "monthlySavings": 47.20
# }

Resource Quota Governance Workflow

Namespace Quota Utilisation Monitoring

# Alert when a namespace is using >80% of its quota
sum by (namespace, resource) (kube_resourcequota{type="used"})
  / sum by (namespace, resource) (kube_resourcequota{type="hard"}) > 0.80

# Find namespaces with NO ResourceQuota (governance gap)
kube_namespace_labels unless on(namespace)
  kube_resourcequota

📝 Knowledge Check