🖊️ kubectl exec — Run Commands in Containers

kubectl exec runs a command inside a running container. It requires the container to have the command binary available — which is why ephemeral containers are needed for distroless images.

# Interactive shell
kubectl exec -it my-app-xyz -- bash
kubectl exec -it my-app-xyz -- sh   # if bash not available

# Run a single command (non-interactive)
kubectl exec my-app-xyz -- env | sort
kubectl exec my-app-xyz -- cat /etc/resolv.conf
kubectl exec my-app-xyz -- wget -qO- http://other-service/healthz

# Target a specific container in a multi-container pod
kubectl exec -it my-app-xyz -c sidecar -- sh

# Test DNS resolution from inside the cluster
kubectl exec -it my-app-xyz -- nslookup kubernetes.default.svc.cluster.local

# Test connectivity to another service
kubectl exec -it my-app-xyz -- nc -zv payments-svc 8080
kubectl exec -it my-app-xyz -- curl -v http://payments-svc:8080/healthz

# Check file permissions and ownership
kubectl exec my-app-xyz -- ls -la /app/config/

📜 kubectl logs — Advanced Patterns

# Current logs, last 100 lines
kubectl logs --tail=100 my-app-xyz

# Previous container instance (after restart)
kubectl logs --previous my-app-xyz

# All containers in a pod (prefix shows which container)
kubectl logs my-app-xyz --all-containers --prefix

# Stream logs from ALL pods matching a selector
kubectl logs -l app=my-app -f --all-containers --prefix --max-log-requests=10

# Since a specific time
kubectl logs my-app-xyz --since-time="2024-01-15T10:30:00Z"
kubectl logs my-app-xyz --since=1h

# Pipe through grep for fast filtering
kubectl logs -l app=my-app --tail=500 | grep -i error
kubectl logs -l app=my-app --tail=500 | grep -v '/healthz'  # remove noise

# Combine with jq for structured JSON logs
kubectl logs my-app-xyz --tail=200 | jq 'select(.level=="error")'
kubectl logs my-app-xyz --tail=200 | jq '{ts: .timestamp, msg: .message, err: .error}'

📋 Quick Reference — Debug Decision Tree

Useful one-liners for production incidents

# Find all pods NOT in Running or Completed state
kubectl get pods -A --field-selector='status.phase!=Running,status.phase!=Succeeded'

# Find pods with high restart counts
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' | tail -20

# Get all events sorted by time (cluster-wide)
kubectl get events -A --sort-by='.lastTimestamp' | tail -30

# Describe every pod in a namespace (batch review)
kubectl describe pods -n production | grep -E 'Name:|State:|Reason:|Message:'

# Watch pod status in real time
kubectl get pods -n production -w

# Copy a file out of a container
kubectl cp my-app-xyz:/app/config/settings.json ./settings-debug.json

# Copy a file INTO a container
kubectl cp ./debug-config.json my-app-xyz:/tmp/debug-config.json

🖥️ kubectl debug node — Node-Level Inspection

Sometimes the problem is at the node level — a failing kubelet, full disk, networking issue, or kernel problem. kubectl debug node creates a privileged pod on the node and mounts the host filesystem, giving you root access to the node without SSH.

# Open a debug shell on node "worker-1"
kubectl debug node/worker-1 \
  --image=nicolaka/netshoot \
  -it -- bash

# Inside the debug pod — host filesystem is at /host
root@node-debugger:/# chroot /host   # enter the host OS

# Check node-level logs (kubelet, containerd)
root@worker-1:/# journalctl -u kubelet -f --no-pager
root@worker-1:/# journalctl -u containerd --since "5 minutes ago"

# Check disk usage (full disk kills the kubelet)
root@worker-1:/# df -h
root@worker-1:/# du -sh /var/lib/containerd/*

# List all running containers on the node
root@worker-1:/# crictl ps

# Inspect a specific container's logs via crictl
root@worker-1:/# crictl logs <container-id>

# Check network interfaces and routing table
root@worker-1:/# ip route
root@worker-1:/# ip addr show
root@worker-1:/# iptables -t nat -L -n | grep KUBE

Inspecting node resource pressure

# Check node conditions (MemoryPressure, DiskPressure, PIDPressure)
kubectl describe node worker-1 | grep -A5 Conditions

# Watch node events
kubectl get events --field-selector involvedObject.name=worker-1 --sort-by='.lastTimestamp'

# Check kubelet eviction thresholds in use
kubectl get --raw /api/v1/nodes/worker-1/proxy/configz | jq '.kubeletconfig.evictionHard'

🔌 kubectl port-forward — Reach Internal Services

kubectl port-forward creates a tunnel from your local machine to any pod or service port inside the cluster. Essential for accessing dashboards, databases, or debugging services not exposed via Ingress.

# Forward local port 8080 → pod port 80
kubectl port-forward pod/my-app-xyz 8080:80

# Forward to a Service (picks a random ready pod endpoint)
kubectl port-forward svc/my-service 8080:80

# Forward to a Deployment
kubectl port-forward deploy/my-app 8080:80

# Multiple ports at once
kubectl port-forward pod/postgres-0 5432:5432 9187:9187

# Bind to all interfaces (access from other machines on LAN)
kubectl port-forward --address 0.0.0.0 svc/grafana 3000:80 -n monitoring

# Quick one-liners for common dashboards
kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring &
kubectl port-forward svc/jaeger-query 16686:16686 -n observability &

🧠 Knowledge Check