🚀 Production HA Controller Deployment

Running 2–3 replicas with leader election gives fast failover without split-brain. Here's a production-grade Deployment for an operator:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-operator
  namespace: my-operator-system
spec:
  replicas: 2    # 2 replicas: 1 active leader + 1 hot standby
  selector:
    matchLabels: { app: my-operator }
  template:
    metadata:
      labels: { app: my-operator }
    spec:
      serviceAccountName: my-operator
      terminationGracePeriodSeconds: 10
      containers:
      - name: manager
        image: ghcr.io/myorg/my-operator:v1.0.0
        args:
        - "--leader-elect=true"
        - "--health-probe-bind-address=:8081"
        - "--metrics-bind-address=:8080"
        livenessProbe:
          httpGet: { path: /healthz, port: 8081 }
          initialDelaySeconds: 15
          periodSeconds:       20
        readinessProbe:
          httpGet: { path: /readyz, port: 8081 }
          initialDelaySeconds: 5
          periodSeconds:       10
        resources:
          requests: { cpu: 100m, memory: 128Mi }
          limits:   { memory: 256Mi }
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          seccompProfile: { type: RuntimeDefault }
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels: { app: my-operator }

Required RBAC for lease management

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-operator-leader-election
  namespace: my-operator-system
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs:     ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["events"]   # for election events
  verbs:     ["create", "patch"]

Graceful leadership handoff

With LeaderElectionReleaseOnCancel: true, when a leader receives SIGTERM (pod shutdown), it releases the lease immediately before exiting — instead of waiting for the lease to expire. This reduces failover time from ~15s to ~2s during planned restarts (rolling updates, node drains):

# Without LeaderElectionReleaseOnCancel:
# 1. Leader receives SIGTERM → starts graceful shutdown
# 2. Standby waits up to LeaseDuration (15s) for lease to expire
# 3. Standby wins election → starts reconciling   (15s gap)

# With LeaderElectionReleaseOnCancel: true:
# 1. Leader receives SIGTERM → releases lease immediately
# 2. Standby sees expired lease → wins election in RetryPeriod (2s)
# 3. Standby starts reconciling                   (2s gap)

Observing leader elections

# Watch the current leader
kubectl get lease my-operator-leader -n my-operator-system -w

# See election history in events
kubectl get events -n my-operator-system \
  --field-selector reason=LeaderElection \
  --sort-by='.lastTimestamp'

# Check which replica is the current leader
kubectl get lease my-operator-leader -n my-operator-system \
  -o jsonpath='{.spec.holderIdentity}'
# my-operator-7d9f8c-xkp2j

# Count leadership transitions (high count = instability)
kubectl get lease my-operator-leader -n my-operator-system \
  -o jsonpath='{.spec.leaseTransitions}'

🏷️ The Lease Object — How Election Works

Kubernetes leader election uses a Lease object in the coordination.k8s.io/v1 API group. The Lease holds the identity of the current leader and a renewal timestamp. Candidates race to acquire the lease by updating it atomically — the API server's optimistic concurrency (resourceVersion) ensures only one wins.

Lease object in etcd

# Inspect a controller's current leader
kubectl get lease my-operator-leader -n my-operator-system -o yaml

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: my-operator-leader
  namespace: my-operator-system
spec:
  acquireTime:          "2024-01-15T09:55:00.000000Z"
  holderIdentity:       my-operator-7d9f8c-xkp2j  # pod name
  leaseDurationSeconds: 15     # how long the lease is valid
  leaseTransitions:     3      # how many times leadership changed
  renewTime:            "2024-01-15T10:00:04.123456Z"

Enabling leader election in controller-runtime

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    Scheme: scheme,

    // Enable leader election
    LeaderElection:                true,
    LeaderElectionID:              "my-operator-leader",        // Lease name
    LeaderElectionNamespace:       "my-operator-system",        // Lease namespace
    LeaderElectionReleaseOnCancel: true,                        // release on graceful shutdown

    // Tuning parameters
    LeaseDuration: func() *time.Duration { d := 15 * time.Second; return &d }(),
    RenewDeadline: func() *time.Duration { d := 10 * time.Second; return &d }(),
    RetryPeriod:   func() *time.Duration { d :=  2 * time.Second; return &d }(),
})

Election parameter tuning

🧠 Knowledge Check