💰 Cost Levers & Typical Savings

Kubernetes clusters are notoriously over-provisioned. Most teams run at 15–30% actual CPU utilisation. The levers below are ordered by typical impact and ease of implementation.

📐 Right-sizing Containers

Over-requesting resources is the most common source of wasted spend. A container requesting 2 CPU cores but using 200m means you're paying for 1.8 cores of idle capacity — multiplied across hundreds of pods.

Finding Over-provisioned Workloads

# See actual vs requested CPU/memory
kubectl top pods -A --sort-by=cpu

# Find pods with no resource requests at all (BestEffort — dangerous)
kubectl get pods -A -o json | jq -r '
  .items[] |
  select(.spec.containers[].resources.requests == null) |
  [.metadata.namespace, .metadata.name] | @tsv'

# Prometheus queries to find over-provisioned containers
# CPU request vs actual usage (last 1 week avg)
# ratio < 0.3 = using less than 30% of what was requested
sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total[1w])
) /
sum by (namespace, pod, container) (
  kube_pod_container_resource_requests{resource="cpu"}
)

# Memory: avg usage vs limit
avg by (namespace, pod, container) (
  container_memory_working_set_bytes
) /
sum by (namespace, pod, container) (
  kube_pod_container_resource_limits{resource="memory"}
)

Right-sizing Rule of Thumb

Vertical Pod Autoscaler (VPA)

VPA automatically recommends (and optionally sets) right-sized CPU and memory requests based on historical usage. In Off mode it only reports recommendations — safe to deploy anywhere.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
  namespace: my-app
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"      # Recommend only — don't restart pods
    # updateMode: "Auto"   # Restart pods with new requests (use in non-prod first)
  resourcePolicy:
    containerPolicies:
      - containerName: app
        minAllowed:
          cpu: "50m"
          memory: "64Mi"
        maxAllowed:
          cpu: "4"
          memory: "4Gi"

# View VPA recommendations
kubectl describe vpa my-app-vpa -n my-app
# Recommendation:
#   Container Recommendations:
#     Container Name: app
#     Lower Bound:    cpu: 80m, memory: 120Mi
#     Target:         cpu: 150m, memory: 200Mi   ← set this as requests
#     Upper Bound:    cpu: 600m, memory: 800Mi

Waste Detection — Idle & Zombie Resources

# Find deployments with 0 replicas (zombie resources)
kubectl get deployments -A -o json | jq -r '
  .items[] | select(.spec.replicas == 0) |
  [.metadata.namespace, .metadata.name] | @tsv'

# Find unattached PVCs (paying for storage with no consumer)
kubectl get pvc -A -o json | jq -r '
  .items[] | select(.status.phase == "Bound") |
  select(.metadata.annotations["volume.kubernetes.io/selected-node"] == null) |
  [.metadata.namespace, .metadata.name, .spec.resources.requests.storage] | @tsv'

# Find LoadBalancer Services with no endpoints (paying for LB with no traffic)
kubectl get svc -A --field-selector=spec.type=LoadBalancer -o json | \
  jq -r '.items[] | [.metadata.namespace, .metadata.name, .status.loadBalancer.ingress[0].ip] | @tsv'

⚡ Spot Nodes & Cluster Autoscaler

Spot (AWS) / Preemptible (GCP) / Spot (Azure) instances offer 60–80% discounts over on-demand pricing. The trade-off: the cloud provider can reclaim them with 2-minute notice. Kubernetes handles this gracefully when workloads are designed for interruption.

Workload Suitability for Spot

Mixed Node Pool Strategy

# Taint spot nodes so only opted-in pods schedule there
kubectl taint nodes -l node-lifecycle=spot \
  node-lifecycle=spot:NoSchedule

# Workloads that tolerate spot
spec:
  tolerations:
    - key: "node-lifecycle"
      operator: "Equal"
      value: "spot"
      effect: "NoSchedule"
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: node-lifecycle
                operator: In
                values: ["spot"]

Cluster Autoscaler Configuration

# Key CA flags (set as args in the Deployment)
--scale-down-enabled=true
--scale-down-delay-after-add=10m       # wait 10m after scale-up before scaling down
--scale-down-unneeded-time=10m         # node idle for 10m before scale-down
--scale-down-utilization-threshold=0.5 # scale down if CPU+mem both < 50%
--max-graceful-termination-sec=600     # 10min for pods to terminate
--balance-similar-node-groups=true     # balance spot pools across AZs
--skip-nodes-with-system-pods=false    # allow scale-down even with system pods

# Prevent CA from scaling down a specific node
kubectl annotate node worker-1 \
  cluster-autoscaler.kubernetes.io/scale-down-disabled=true

# Check CA activity log
kubectl -n kube-system logs deployment/cluster-autoscaler | grep -i "scale\|removed\|added"

Handling Spot Interruptions Gracefully

# AWS Node Termination Handler — catches spot interruption notices
# and cordons/drains the node before it's reclaimed
helm install aws-node-termination-handler \
  aws/aws-node-termination-handler \
  --namespace kube-system \
  --set enableSpotInterruptionDraining=true \
  --set enableScheduledEventDraining=true \
  --set nodeTerminationGracePeriod=120

# PodDisruptionBudget — ensures ≥1 pod stays up during spot eviction
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: my-app
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-app

🚀 Karpenter & Namespace Cost Controls

Karpenter vs Cluster Autoscaler

Karpenter NodePool

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]   # prefer spot, fallback to on-demand
        - key: node.kubernetes.io/instance-type
          operator: In
          values:                         # allow many types for spot diversity
            - m5.xlarge
            - m5a.xlarge
            - m5d.xlarge
            - m4.xlarge
            - m6i.xlarge
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: default
  limits:
    cpu: "1000"          # max total CPU across all Karpenter nodes
    memory: "4000Gi"
  disruption:
    consolidationPolicy: WhenUnderutilized   # replace/remove underutilised nodes
    consolidateAfter: 30s

Namespace ResourceQuota for Cost Control

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-alpha-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "20"           # total CPU requests across all pods
    requests.memory: "40Gi"
    limits.cpu: "40"
    limits.memory: "80Gi"
    count/persistentvolumeclaims: "20"
    requests.storage: "500Gi"
    count/services.loadbalancers: "2"   # limit expensive LB services

Scale-to-Zero for Non-Prod

# Use KEDA ScaledObject with minReplicaCount: 0 to scale idle workloads to zero
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-app
  namespace: staging
spec:
  scaleTargetRef:
    name: my-app
  minReplicaCount: 0    # scale to zero when idle
  maxReplicaCount: 10
  cooldownPeriod: 300   # 5 min idle before scale-down to zero
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: http_requests_total
        threshold: "10"
        query: sum(rate(http_requests_total{namespace="staging"}[1m]))

📝 Knowledge Check