GitHub goes down. Your AKS cluster is deleted. ArgoCD state is lost. As an architect, you must plan for these scenarios BEFORE they happen. This lesson covers what can break and how to recover.

Failure Scenarios & Recovery

ScenarioImpactRecovery StrategyPrevention
GitHub outage Can't push, can't run CI Wait (or mirror to GitLab) Git mirror as backup
AKS cluster deleted App down, data lost Recreate cluster → point ArgoCD at Git → full restore IaC (Terraform/Bicep) for cluster
ACR down/deleted Can't pull images Rebuild + push from source Geo-replication (Premium ACR)
ArgoCD lost No sync, drift undetected Reinstall + re-apply Application YAMLs from Git ArgoCD HA mode, backup Application CRDs
Secrets compromised Full breach potential Rotate ALL secrets immediately OIDC everywhere, short-lived tokens, audit logs
Bad deploy to prod Users affected git revert → ArgoCD syncs → rollback Canary + automated analysis

Why GitOps Makes DR Easier

Without GitOps: Cluster dies 1. Recreate cluster (IaC) 2. Which version was deployed? Check CI logs? 3. Re-run pipelines in correct order? Hours to days of recovery 😰 With GitOps: Cluster dies 1. Recreate cluster (IaC) 2. Install ArgoCD 3. Point at GitOps repo → EVERYTHING restores Minutes to restore ✅
GitOps = your cluster is disposable. The ENTIRE desired state lives in Git. Lose the cluster? Create a new one, install ArgoCD, point at Git. Every application, every config, every policy syncs automatically. This is the ultimate DR story.

DR Checklist for CI/CD Architects

  1. ☐ Cluster provisioning is IaC (Terraform/Bicep) — stored in Git, runnable in minutes
  2. ☐ ALL K8s state is in GitOps repo (no manual kubectl apply in production, ever)
  3. ☐ ACR has geo-replication enabled (images survive region failure)
  4. ☐ Secrets are in Azure Key Vault (not in cluster — survives cluster deletion)
  5. ☐ ArgoCD Application definitions are in Git (can reinstall + restore)
  6. ☐ Database backups are automated and tested (restore tested quarterly)
  7. ☐ DR runbook documented and rehearsed

🧠 Recall Check

  1. If your AKS cluster is deleted, what three steps restore everything with GitOps?
  2. Why is GitOps inherently better for DR than push-based CD?
  3. What's the ONE thing that can't be in Git and must be backed up separately?
Reveal answers
  1. 1) Recreate cluster (IaC), 2) Install ArgoCD, 3) Apply root Application (points at GitOps repo) → all apps auto-sync.
  2. With GitOps, the desired state is declared in Git (not scattered across CI logs and manual commands). Recovery = "make actual match desired" which is what ArgoCD does automatically. With push-based, you'd need to figure out which pipeline ran last and re-run them in order.
  3. Data (database contents, persistent volumes). Infrastructure and config are in Git, but stateful data needs separate backup/restore procedures.

Next lesson: Release Pipelines & Semantic Versioning — the professional way to ship versions.