🔧 Worker Node Upgrade Strategies

In-Place Worker Node Upgrade (kubeadm)

# For each worker node (repeat for each):

# 1. Cordon the node (stop new pods being scheduled here)
kubectl cordon worker-1

# 2. Drain (evict all pods gracefully)
kubectl drain worker-1 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=300s

# Wait for drain to complete — check no pods remain except DaemonSets
kubectl get pods -o wide --field-selector spec.nodeName=worker-1

# 3. SSH to the node and upgrade packages
ssh worker-1
apt-mark unhold kubeadm kubelet kubectl
apt-get install -y \
  kubeadm=1.29.3-1.1 \
  kubelet=1.29.3-1.1 \
  kubectl=1.29.3-1.1
kubeadm upgrade node
systemctl daemon-reload
systemctl restart kubelet

# 4. Back on control plane — uncordon
kubectl uncordon worker-1

# 5. Verify node is Ready and new version
kubectl get node worker-1
# NAME       STATUS   ROLES    VERSION
# worker-1   Ready    <none>   v1.29.3

# 6. Repeat for next worker node

Blue-Green Node Pool (Cloud)

# AWS EKS / GKE / AKS approach:

# 1. Create new node group with target version
eksctl create nodegroup \
  --cluster my-cluster \
  --name workers-v129 \
  --kubernetes-version 1.29 \
  --nodes 5

# 2. Cordon all old nodes (prevent new scheduling)
kubectl cordon -l eks.amazonaws.com/nodegroup=workers-v128

# 3. Drain old nodes one at a time
for node in $(kubectl get nodes -l eks.amazonaws.com/nodegroup=workers-v128 -o name); do
  kubectl drain $node --ignore-daemonsets --delete-emptydir-data --force
done

# 4. Delete old node group after pods migrated
eksctl delete nodegroup --cluster my-cluster --name workers-v128

Handling Stuck Drains

# Check why drain is stuck
kubectl get pdb -A
kubectl get pods -o wide --field-selector spec.nodeName=worker-1

# PDB is blocking — check which PDB
kubectl describe pdb my-app-pdb
# Disruptions Allowed: 0  ← all replicas on one node, PDB prevents eviction

# Fix: scale up the deployment first so PDB allows disruption
kubectl scale deployment my-app --replicas=3

# Or temporarily patch the PDB (use with caution)
kubectl patch pdb my-app-pdb -p '{"spec":{"minAvailable":0}}'
# ... drain completes ...
kubectl patch pdb my-app-pdb -p '{"spec":{"minAvailable":1}}'

⏪ Rollback & Add-on Upgrades

Control-Plane Rollback

Kubernetes does not support downgrading the control plane via kubeadm. The only rollback path is restoring from the pre-upgrade etcd snapshot to a fresh cluster running the old version.

# Rollback procedure (last resort):
# 1. Stop kube-apiserver on all control-plane nodes
# 2. Restore etcd from pre-upgrade snapshot (see lesson 83)
# 3. Reinstall old kubeadm/kubelet/kubectl packages
# 4. Re-run kubeadm init with old version config

# This is why pre-upgrade etcd snapshots are MANDATORY.
# Without a snapshot, rollback is impossible.

Add-on Compatibility Matrix

Upgrade Automation with Managed Kubernetes

# EKS — upgrade control plane then node groups
aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version 1.29

# Wait for control plane upgrade
aws eks wait cluster-active --name my-cluster

# Upgrade managed node group
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name workers \
  --kubernetes-version 1.29

# GKE — single command upgrades control plane + node pools
gcloud container clusters upgrade my-cluster \
  --master --cluster-version 1.29

gcloud container clusters upgrade my-cluster \
  --node-pool default-pool \
  --cluster-version 1.29

📝 Knowledge Check