♻️ The Reconciliation Pattern

Every Kubernetes controller implements the same pattern: Observe → Diff → Act. Controllers never push commands at the cluster; instead they continuously compare desired state (from the API server) with actual state (from the cluster or cloud) and apply the minimum change needed to close the gap.

This makes controllers idempotent and self-healing — re-running the same reconcile loop has no side-effects if the state is already correct.

🎛️ Key Built-in Controllers

Deployment Controller — Rolling Update Flow

When you change a Deployment's pod template the Deployment controller executes a rolling update through ReplicaSets:

# Watch what the Deployment controller does during a rollout
kubectl rollout status deployment/nginx --watch

# Under the hood:
# 1. Deployment controller creates a new ReplicaSet (rs-v2) with replicas=0
# 2. It scales rs-v2 up by maxSurge (default 25%) and rs-v1 down by maxUnavailable
# 3. Loops until rs-v2 = desired replicas and rs-v1 = 0
# 4. Old ReplicaSet is retained (for rollback) but scaled to 0

# Check ReplicaSet history
kubectl get replicasets -l app=nginx
# NAME              DESIRED   CURRENT   READY   AGE
# nginx-7d8f9c      3         3         3       5m    ← new
# nginx-5b6c7d      0         0         0       2h    ← old (kept for rollback)

# Rollback uses the old ReplicaSet directly — instant
kubectl rollout undo deployment/nginx

GarbageCollector Controller

The GarbageCollector runs a background graph of owner references. When an owner is deleted, it cascades deletion to all dependents — this is how deleting a Deployment also deletes its ReplicaSets and Pods.

# OwnerReferences — set automatically by controllers
kubectl get pod nginx-7d8f9c-abc -o jsonpath='{.metadata.ownerReferences}'
# [{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"nginx-7d8f9c",
#   "uid":"...","controller":true,"blockOwnerDeletion":true}]

# Foreground deletion: owner stays until all dependents are gone
kubectl delete deployment nginx --cascade=foreground

# Background deletion (default): owner deleted first, GC cleans up dependents async
kubectl delete deployment nginx --cascade=background

# Orphan: delete owner but keep dependents
kubectl delete deployment nginx --cascade=orphan

NodeLifecycle Controller — Pod Eviction

When a node goes unreachable, the NodeLifecycle controller adds a node.kubernetes.io/not-ready taint. After the pod eviction timeout (default 5 minutes), it evicts all Pods on that node. This is why pods reappear on other nodes after a node failure.

# Key flags controlling eviction timing
--node-monitor-grace-period=40s      # time before node marked NotReady
--pod-eviction-timeout=5m0s          # (deprecated) grace period before eviction
# Modern: controlled by node taint tolerations on pods
# Default toleration: 300s for not-ready / unreachable taints

⚙️ Work Queues, Leader Election & Rate Limiting

Work Queue Internals

Each controller uses a rate-limited work queue from client-go/util/workqueue. Events from the informer (watch cache) are enqueued as keys (namespace/name). The reconciler dequeues keys and processes them — deduplication means multiple rapid changes to the same object produce a single reconcile.

// Simplified work queue pattern used by all controllers
func (c *Controller) Run(workers int, stopCh <-chan struct{}) {
    defer c.queue.ShutDown()
    // Start informers
    go c.podInformer.Run(stopCh)
    // Wait for cache sync
    cache.WaitForCacheSync(stopCh, c.podInformer.HasSynced)
    // Start N worker goroutines
    for i := 0; i < workers; i++ {
        go wait.Until(c.runWorker, time.Second, stopCh)
    }
    <-stopCh
}

func (c *Controller) runWorker() {
    for c.processNextItem() {}
}

func (c *Controller) processNextItem() bool {
    key, quit := c.queue.Get()
    if quit { return false }
    defer c.queue.Done(key)

    err := c.reconcile(key.(string))
    if err != nil {
        // Exponential back-off re-queue
        c.queue.AddRateLimited(key)
    }
    return true
}

Leader Election

In HA control planes, multiple kube-controller-manager instances run simultaneously. Only the leader runs the reconcile loops — the others are hot-standby. Leadership is held via a Lease object in the kube-system namespace, renewed every few seconds.

# View the current leader lease
kubectl get lease kube-controller-manager -n kube-system -o yaml
# spec:
#   holderIdentity: master-1_abc-uuid   ← current leader
#   leaseDurationSeconds: 15
#   renewTime: "2024-01-15T12:34:56Z"
#   acquireTime: "2024-01-15T10:00:00Z"

# If the leader fails to renew within leaseDurationSeconds,
# a standby instance acquires the lease and becomes leader
# (typical failover: 15–30 seconds)

Key kube-controller-manager Flags

# Concurrency / workers per controller
--concurrent-deployment-syncs=5     # default 5 deployment workers
--concurrent-replicaset-syncs=5
--concurrent-endpoint-syncs=5
--concurrent-gc-syncs=20            # GarbageCollector workers

# Node eviction / health
--node-monitor-grace-period=40s     # wait before marking node NotReady
--node-monitor-period=5s            # how often to check node status

# Leader election
--leader-elect=true
--leader-elect-lease-duration=15s
--leader-elect-renew-deadline=10s
--leader-elect-retry-period=2s

# HPA
--horizontal-pod-autoscaler-sync-period=15s
--horizontal-pod-autoscaler-tolerance=0.1

# Namespace lifecycle
--namespace-sync-period=5m0s

📊 Observability & Production Tips

Key Metrics

Common Issues & Fixes

# Issue: Deployment stuck in rollout — pods not becoming ready
kubectl rollout status deployment/myapp
# Waiting for deployment "myapp" rollout to finish: 1 out of 3 new replicas updated

# Diagnose: check events and pod status
kubectl describe deployment myapp
kubectl get pods -l app=myapp --sort-by='.status.startTime'
kubectl describe pod myapp-<new-rs>-<hash>

# Issue: Pods not evicted after node failure (5min default delay)
# Speed up for testing — reduce toleration on pods
spec:
  tolerations:
    - key: "node.kubernetes.io/not-ready"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 30   # evict after 30s instead of 300s

# Issue: HPA not scaling
kubectl describe hpa myapp-hpa
# Check: is metrics-server running?
kubectl top pods -n my-namespace
# Check: are resource requests set on pods? (HPA requires them)

📝 Knowledge Check