🗑️ Eviction Diagnosis

Pods are evicted when the kubelet detects node pressure — low memory, low disk, or too many processes. Eviction is a normal protective mechanism but becomes a problem when it's continuous or targets critical workloads.

Identifying evictions

# Evicted pods stay in the namespace with STATUS=Evicted
kubectl get pods -A | grep Evicted

# See why a pod was evicted
kubectl describe pod evicted-pod-xyz | grep -A 5 "Status:"
# Status:   Failed
# Reason:   Evicted
# Message:  The node was low on resource: memory.
#           Threshold quantity: 100Mi, available: 47Mi

# Check current node conditions
kubectl describe node worker-1 | grep -A 15 Conditions
# MemoryPressure   True   → node actively evicting
# DiskPressure     True   → disk full, kubelet will evict
# PIDPressure      True   → too many processes

# Clean up completed/evicted pods (they accumulate)
kubectl delete pods -A --field-selector='status.phase=Failed'

Eviction prevention strategies

💥 OOMKilled

OOMKilled (exit code 137) means the Linux kernel's OOM killer terminated the container because it exceeded its memory limit. Unlike eviction (node-level), OOMKill is container-level — enforced by cgroups.

# Identify OOMKilled containers
kubectl get pods -A -o json | jq '.items[] | select(
  .status.containerStatuses[]?.lastState.terminated.reason == "OOMKilled"
) | {name: .metadata.name, ns: .metadata.namespace}'

# Check memory usage vs limit for a running pod
kubectl top pod my-app-xyz --containers

# See the limit that was exceeded
kubectl get pod my-app-xyz -o jsonpath='{.spec.containers[0].resources.limits.memory}'

# Check container memory usage in the describe output
kubectl describe pod my-app-xyz | grep -A 10 "Last State"
# Last State: Terminated
#   Reason:    OOMKilled
#   Exit Code: 137

🧭 Universal Troubleshooting Methodology

Every Kubernetes problem follows the same diagnostic ladder. Work top-down, stop when you find the cause:

1. kubectl get — confirm the resource exists and note its phase/status/age/restarts
2. kubectl describe — read the Events section at the bottom; this is the most useful signal
3. kubectl logs [--previous] — read application output; always check previous on restarts
4. kubectl get endpoints / pvc / configmap — verify dependencies exist and are populated
5. kubectl exec — test connectivity — nslookup, curl, nc from inside the affected pod
6. kubectl debug / node — ephemeral container or node-level inspection for deep issues
7. Metrics + logs — check Prometheus for resource saturation, Loki for correlated log spikes

Quick-reference: status → most likely cause

🔵 Pending Pods

Pending means the scheduler could not find a suitable node. The pod exists in etcd but no node has been assigned. Always check the Events section of kubectl describe pod first — the scheduler emits a clear reason.

Common Pending causes and fixes

# Full diagnostic sequence for a Pending pod

# 1. Read the scheduler reason
kubectl describe pod my-app-xyz | grep -A 10 Events

# 2. Check allocatable resources on all nodes
kubectl describe nodes | grep -A 5 "Allocated resources"

# 3. Check if the PVC is bound (if storage-related)
kubectl get pvc -n production
kubectl describe pvc my-pvc -n production

# 4. Test scheduling manually with a dry-run
kubectl run test-sched --image=nginx --dry-run=server -o yaml \
  --overrides='{"spec":{"nodeSelector":{"disktype":"ssd"}}}' 2>&1

🌐 DNS Failures

DNS failures are subtle — they appear as connection timeouts or "host not found" errors in application logs, often misdiagnosed as a service being down when the service is perfectly healthy.

DNS resolution hierarchy in Kubernetes

# A pod's /etc/resolv.conf looks like:
nameserver 10.96.0.10      # CoreDNS ClusterIP
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

# Resolution order for "payments-svc":
# 1. payments-svc.default.svc.cluster.local   ← found! stops here
# 2. payments-svc.svc.cluster.local
# 3. payments-svc.cluster.local
# 4. payments-svc. (external DNS)

Diagnostic workflow

# Step 1: test DNS from inside a pod
kubectl exec -it my-app-xyz -- nslookup kubernetes.default.svc.cluster.local
# If this fails → CoreDNS is the problem
# If this works but service DNS fails → check the service/endpoints

# Step 2: check CoreDNS is healthy
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

# Step 3: verify the Service and its Endpoints exist
kubectl get svc payments-svc -n production
kubectl get endpoints payments-svc -n production
# Empty Endpoints → no pods matching the Service selector are Ready

# Step 4: check ndots — external hostnames with few dots get search suffixes appended
# "api.stripe.com" has 2 dots < ndots:5, so 5 failed local lookups happen first!
# Fix: use trailing dot "api.stripe.com." OR set dnsConfig.options.ndots: 2
spec:
  dnsConfig:
    options:
    - name: ndots
      value: "2"   # reduces unnecessary local DNS queries

🧠 Knowledge Check