🚨 First-Responder Triage Framework

When an alert fires, follow a structured funnel: Scope → Locate → Diagnose → Mitigate → Fix → Post-mortem. Resist the urge to make changes without first understanding the blast radius.

🔔 Alert 📐 Scope 🔍 Locate 🧬 Diagnose 🛡️ Mitigate 📋 Fix + PIR PagerDuty 1 pod? all pods? which node/ns? logs, events, metrics rollback / cordon root cause + docs

Universal First Commands

# Cluster-wide health snapshot — always start here
kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get events -A --sort-by='.lastTimestamp' | tail -40

# Check component health
kubectl get componentstatuses   # deprecated but still useful
kubectl -n kube-system get pods | grep -v Running

# Quick node resource view
kubectl top nodes
kubectl top pods -A --sort-by=memory | head -20

# Recent events for a namespace
kubectl get events -n my-app --sort-by='.lastTimestamp'

# Who changed what recently? (audit log shortcut)
kubectl get events -A --sort-by='.lastTimestamp' \
  | grep -i "Warning\|Error\|Failed\|BackOff"

🐛 Pod & Node Failure Runbooks

🔴 Pod CrashLoopBackOff

  1. Check restart count and exit code: kubectl describe pod <name> | grep -A5 "Last State"
  2. Read current logs: kubectl logs <pod> --tail=100
  3. Read previous container logs: kubectl logs <pod> --previous --tail=100
  4. Check events: kubectl get events --field-selector involvedObject.name=<pod>
  5. Exit code 1 → application error; exit code 137 → OOMKilled; exit code 139 → segfault
  6. Override command to debug: patch with command: ["sleep","infinity"], then exec in

🧠 Pod OOMKilled (exit code 137)

  1. Confirm: kubectl describe pod <name> | grep -i "OOMKilled"
  2. Check actual usage vs limit in Prometheus: container_memory_working_set_bytes
  3. Get current limits: kubectl get pod <name> -o jsonpath='{.spec.containers[*].resources}'
  4. If limit too low → increase resources.limits.memory
  5. If app leaks memory → profile + fix app; set requests == limits for Guaranteed QoS
  6. Check namespace LimitRange: kubectl get limitrange -n <ns>

⏳ Pod Stuck Pending

  1. Check events: kubectl describe pod <name> | grep -A20 Events
  2. "Insufficient CPU/memory" → no capacity; scale cluster or reduce requests
  3. "No nodes match selector" → fix nodeSelector or label a node
  4. "Unschedulable: taint" → add toleration or remove taint
  5. "PVC not bound" → kubectl get pvc -n <ns>; check StorageClass
  6. Check scheduler is running: kubectl -n kube-system get pods | grep scheduler

⚠️ Node NotReady

  1. Check conditions: kubectl describe node <name> | grep -A20 Conditions
  2. SSH and check kubelet: systemctl status kubelet
  3. Kubelet logs: journalctl -u kubelet -n 100 --no-pager
  4. Check disk full: df -h → prune images: crictl rmi --prune
  5. Check containerd: systemctl status containerd; restart if crashed
  6. If unrecoverable: kubectl cordon <node> && kubectl drain <node> --ignore-daemonsets

Pod Debugging Toolkit

# Ephemeral debug container (K8s 1.23+)
kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container>

# Copy crashing pod with a shell override
kubectl debug <pod> -it --copy-to=debug-pod \
  --image=<same-image> --container=<container> -- /bin/sh

# Real-time resource usage
kubectl top pod <name> --containers

# Port-forward to test directly
kubectl port-forward pod/<name> 8080:8080

# Check env vars in a pod
kubectl exec <pod> -- env | sort

🏛️ Control-Plane & Networking Incidents

🏛️ API Server Unavailable / kubectl Timeout

  1. Check if LB is healthy: curl -k https://<LB-VIP>:6443/healthz
  2. Check each apiserver pod directly: curl -k https://<CP-IP>:6443/healthz
  3. Check control-plane pods: ssh cp-1 "crictl ps | grep kube-api"
  4. Check apiserver logs: ssh cp-1 "crictl logs <apiserver-container-id> 2>&1 | tail -50"
  5. Check etcd health: etcdctl endpoint health --cluster ...
  6. If etcd is unhealthy → etcd quorum lost; restore from snapshot (lesson 83)
  7. If apiserver is up but unresponsive → check etcd_request_duration_seconds; etcd may be overloaded
  8. Mitigation: if one CP node is bad, remove it from LB target group temporarily

🌐 Service DNS Resolution Failing

  1. Test from a pod: kubectl run dns-test --image=busybox --rm -it -- nslookup kubernetes
  2. Check CoreDNS pods: kubectl -n kube-system get pods -l k8s-app=kube-dns
  3. Check CoreDNS logs: kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50
  4. Check CoreDNS config: kubectl -n kube-system get configmap coredns -o yaml
  5. Test ClusterIP directly to bypass DNS: curl http://<ClusterIP>:<port>
  6. If ClusterIP works but DNS fails → CoreDNS issue; restart CoreDNS pods
  7. Check kube-dns Service: kubectl -n kube-system get svc kube-dns — IP must match cluster DNS IP
  8. Check node's resolv.conf: cat /etc/resolv.conf on failing pod's node

🔌 Service Not Reachable (ClusterIP)

  1. Check endpoints are populated: kubectl get endpoints <svc-name> -n <ns>
  2. If endpoints are empty → pods are not Ready; check readiness probe failures
  3. Check label selector matches pod labels: kubectl get svc <name> -o yaml | grep selector
  4. Check pod labels: kubectl get pods -n <ns> --show-labels
  5. Verify kube-proxy is running: kubectl -n kube-system get pods -l k8s-app=kube-proxy
  6. Check iptables rules exist: iptables -t nat -L KUBE-SERVICES -n | grep <ClusterIP>
  7. Check NetworkPolicy is not blocking: kubectl get networkpolicy -n <ns>

Network Connectivity Test Matrix

# Run from a debug pod in the affected namespace
kubectl run netshoot --image=nicolaka/netshoot --rm -it -- bash

# Pod → Service (DNS)
curl -v http://my-svc.my-ns.svc.cluster.local

# Pod → Pod direct
curl -v http://10.0.2.11:8080

# Pod → External
curl -v https://api.example.com

# Check traceroute
traceroute 10.96.0.1    # ClusterIP of kubernetes service

# Check conntrack entries
conntrack -L | grep 10.96.0.10

# Test NodePort from outside
curl -v http://<node-IP>:<nodePort>

💾 Storage & Performance Incidents

💾 PVC Stuck in Pending

  1. Check PVC status: kubectl describe pvc <name> -n <ns>
  2. "no persistent volumes available" → no matching PV; check StorageClass provisioner
  3. Check StorageClass: kubectl get sc — is the right one the default?
  4. Check CSI provisioner pods: kubectl -n kube-system get pods | grep csi
  5. Check CSI provisioner logs for the failing PVC name
  6. "waiting for first consumer" → volumeBindingMode: WaitForFirstConsumer; PVC binds only when pod is scheduled — this is normal
  7. Check VolumeAttachment if PV exists but won't mount: kubectl get volumeattachments

📈 High Latency / Performance Degradation

  1. Check if the issue is cluster-wide or isolated: kubectl top nodes
  2. Identify CPU/memory hot spots: kubectl top pods -A --sort-by=cpu | head -20
  3. Check for CPU throttling: Prometheus container_cpu_cfs_throttled_seconds_total
  4. Check HPA status: kubectl get hpa -A — is it failing to scale?
  5. Check Cluster Autoscaler: kubectl -n kube-system logs deployment/cluster-autoscaler | tail -30
  6. Check API server latency: Prometheus apiserver_request_duration_seconds p99
  7. Check etcd latency: etcd_disk_wal_fsync_duration_seconds p99 > 10ms is a red flag
  8. Check for noisy neighbours: identify pods without requests/limits: kubectl get pods -A -o json | jq '.items[] | select(.spec.containers[].resources.requests == null) | .metadata.name'

Quick Incident Reference

SymptomFirst checkLikely cause
Pod CrashLoopBackOffkubectl logs --previousApp error, OOMKill, bad config
Pod Pending foreverkubectl describe pod eventsInsufficient resources, taint, PVC unbound
Node NotReadyjournalctl -u kubeletDisk full, containerd crash, network partition
DNS resolution failskubectl logs on CoreDNS podsCoreDNS crash, wrong configmap, networkpolicy
Service not reachablekubectl get endpointsNo ready pods, label mismatch, kube-proxy down
PVC stuck Pendingkubectl describe pvcNo matching PV, CSI driver error, wrong SC
API server slowetcd latency metricsetcd overloaded, disk pressure, leader election
kubectl connection refusedcurl -k https://<LB>:6443/healthzLB down, all apiservers down
High pod restart rateLiveness probe eventsProbe misconfigured, app health endpoint broken
Pods not evicted from dead nodeNode taint + pod tolerationstolerationSeconds too high, node not tainted yet

📝 Knowledge Check

Q1. A pod has status OOMKilled and keeps restarting. kubectl describe pod shows exit code 137. What is the immediate fix and longer-term solution?
  • A) Restart the node — OOMKills are caused by node-level memory pressure
  • B) Increase resources.limits.memory immediately; long-term profile the app for memory leaks
  • C) Remove the memory limit entirely so the pod can use as much as needed
  • D) Scale the deployment to more replicas to spread memory usage
✅ B) Increase limit; then profile. Exit code 137 = SIGKILL from the Linux OOM killer, triggered when the container exceeds its limits.memory. Immediate fix: increase the limit. Long-term: profile with memory analysis tools to find leaks. Removing limits entirely (C) is dangerous — it makes the pod BestEffort QoS and risks killing other pods on the node.
Q2. kubectl get endpoints my-svc returns <none>. Pods for the service exist and are Running. What is the most likely cause?
  • A) kube-proxy is not running on the nodes
  • B) The Service's label selector does not match the pods' labels
  • C) The pods don't have resource limits set
  • D) CoreDNS is not resolving the service name
✅ B) Label selector mismatch. The EndpointSlice controller builds endpoints from pods whose labels match the Service's spec.selector. If pods are Running but endpoints are empty, the selector doesn't match. Compare kubectl get svc my-svc -o yaml | grep -A5 selector with kubectl get pods --show-labels.
Q3. All kubectl commands return "connection refused" but you can SSH to all control-plane nodes and the apiserver pods are Running. What should you check first?
  • A) Restart the kube-apiserver pods on all control-plane nodes
  • B) Check the load balancer in front of the API server — it may be unhealthy or misconfigured
  • C) Check etcd quorum — the apiserver won't serve if etcd is down
  • D) Re-generate the kubeconfig with a new certificate
✅ B) Check the load balancer. kubectl connects to the LB VIP/DNS (from kubeconfig). If apiserver pods are Running but kubectl gets "connection refused", the LB is the broken component — health checks may have marked all backends unhealthy, or the LB itself crashed. Test by curling the LB endpoint directly: curl -k https://<LB-VIP>:6443/healthz.