🎯 DR Strategies — The Four Tiers

Choose your DR strategy based on your RTO/RPO requirements and budget. Each tier roughly doubles cost while halving recovery time.

RTO & RPO Definitions

💾 What to Back Up & How Often

Automated etcd Backup CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  schedule: "0 * * * *"    # every hour
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          nodeName: cp-1               # run on first control-plane node
          containers:
            - name: etcd-backup
              image: bitnami/etcd:3.5
              command:
                - /bin/sh
                - -c
                - |
                  ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M).db \
                    --endpoints=https://127.0.0.1:2379 \
                    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
                    --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
                    --key=/etc/kubernetes/pki/apiserver-etcd-client.key
                  aws s3 cp /backup/etcd-$(date +%Y%m%d-%H%M).db \
                    s3://my-etcd-backups/$(date +%Y/%m/%d)/ \
                    --storage-class STANDARD_IA
              volumeMounts:
                - name: etcd-certs
                  mountPath: /etc/kubernetes/pki/etcd
                  readOnly: true
          volumes:
            - name: etcd-certs
              hostPath:
                path: /etc/kubernetes/pki/etcd
          restartPolicy: OnFailure
          tolerations:
            - key: node-role.kubernetes.io/control-plane
              effect: NoSchedule

Velero Scheduled Backup for PV Data

velero schedule create hourly-production \
  --schedule="@every 1h" \
  --include-namespaces production,staging \
  --ttl 168h \
  --storage-location default

# Monitor backup health
velero backup get
velero schedule get

# Alert if latest backup is older than 2 hours (Prometheus)
# time() - velero_backup_last_successful_timestamp > 7200

🌍 Multi-Region & Cluster Failover

Active-Active Multi-Cluster with Submariner

# Submariner creates an encrypted overlay between clusters
# allowing cross-cluster service discovery and pod-to-pod routing

# Install on both clusters
subctl deploy-broker --kubeconfig primary.kubeconfig
subctl join broker-info.subm \
  --kubeconfig secondary.kubeconfig \
  --clusterid secondary

# Export a service to be discoverable from other clusters
kubectl apply -f - <<EOF
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: my-api
  namespace: production
EOF

# From secondary cluster, discover the service
kubectl get serviceimport -n production
# Access via: my-api.production.svc.clusterset.local

Cluster Failover Runbook (Backup & Restore tier)

1. Declare incident — alert on-call team, open incident channel, assign incident commander
2. Assess primary cluster — determine if recoverable or if full DR activation is needed
3. Point DNS to DR region — update Route53/Cloud DNS health-check target or lower TTL to failover
4. Provision DR cluster — use Terraform/Pulumi IaC; cluster should provision in 5–10 min
5. Restore etcd snapshot — etcdctl snapshot restore from latest S3 backup (see lesson 83)
6. Bootstrap GitOps — install ArgoCD, apply root Application; manifests auto-apply from Git
7. Restore PV data — velero restore create --from-backup <latest>
8. Validate services — run smoke tests against DR endpoints, check database connectivity
9. Communicate status — update status page, notify stakeholders, estimated recovery time
10. Post-incident — write PIR (Post-Incident Review), update runbook with lessons learned

💥 Chaos Engineering & DR Testing

An untested DR plan is not a DR plan. Chaos engineering deliberately injects failures in controlled conditions to validate that your system handles them gracefully — and that your runbooks actually work.

LitmusChaos — Kubernetes-Native Chaos

helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm
helm install chaos litmuschaos/litmus \
  --namespace litmus \
  --create-namespace

# Pod Delete experiment — kills random pods in a namespace
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-delete-chaos
  namespace: production
spec:
  appinfo:
    appns: production
    applabel: "app=my-api"
    appkind: deployment
  chaosServiceAccount: pod-delete-sa
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"      # run for 60 seconds
            - name: CHAOS_INTERVAL
              value: "10"      # delete a pod every 10s
            - name: FORCE
              value: "false"   # graceful termination
        probe:
          - name: check-availability
            type: httpProbe
            httpProbe/inputs:
              url: https://my-api.example.com/health
              insecureSkipVerify: false
              responseTimeout: 5000
            runProperties:
              probeTimeout: 10
              interval: 2
              attempt: 10

DR Testing Schedule

Key DR Metrics to Track

📝 Knowledge Check