Kubernetes Secrets solve the "how do I inject config into Pods" problem, but they don't solve the "where do secrets live and who manages their lifecycle" problem. In production, secrets live in dedicated secret management systems — Vault, AWS Secrets Manager, GCP Secret Manager — and are synced into Kubernetes. This lesson covers the ecosystem and patterns.

1. Why External Secrets Management?

K8s native Secrets have fundamental limitations for production:

The Solution Landscape

2. External Secrets Operator (ESO)

The most popular approach. ESO is a K8s operator that syncs secrets from external stores into native K8s Secrets. It uses Custom Resources to declare what to sync.

Architecture

# 1. SecretStore — HOW to connect to the external provider
# 2. ExternalSecret — WHAT to fetch and WHERE to put it
# 3. Result: a native K8s Secret (created/updated by ESO controller)

SecretStore (or ClusterSecretStore)

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-service-account   # Uses IRSA (IAM Roles for SAs)

ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h              # ← How often to sync from external store
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-creds                 # ← K8s Secret to create
    creationPolicy: Owner          # ESO owns (deletes when ExternalSecret deleted)
  data:
    - secretKey: username          # ← key in the K8s Secret
      remoteRef:
        key: prod/database         # ← path in AWS Secrets Manager
        property: username         # ← JSON field within the secret
    - secretKey: password
      remoteRef:
        key: prod/database
        property: password

What Happens

1. ESO controller reads the ExternalSecret CR
2. Connects to AWS Secrets Manager using the SecretStore credentials
3. Fetches prod/database and extracts username and password
4. Creates (or updates) K8s Secret db-creds in namespace production
5. Every refreshInterval, re-syncs (catches rotations in the external store)

Supported Providers

3. Alternative Approaches

Sealed Secrets (Bitnami)

Encrypt Secrets client-side so they can be stored in Git safely.

# Install kubeseal CLI + controller in cluster
# Encrypt a Secret:
kubectl create secret generic db-creds \
  --from-literal=password=s3cr3t \
  --dry-run=client -o yaml | kubeseal \
  --controller-name=sealed-secrets \
  --controller-namespace=kube-system \
  -o yaml > sealed-secret.yaml

# sealed-secret.yaml can be committed to Git safely
# The SealedSecret controller decrypts it inside the cluster
# and creates a regular K8s Secret

Secrets Store CSI Driver

Mounts secrets from external stores directly into Pods as volumes — bypasses K8s Secrets entirely.

# SecretProviderClass defines what to fetch:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.company.com"
    roleName: "web-app"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/db"
        secretKey: "password"

# Pod mounts it:
spec:
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: vault-db-creds
  containers:
    - volumeMounts:
        - name: secrets
          mountPath: /mnt/secrets
          readOnly: true

SOPS + GitOps (Flux/ArgoCD)

Encrypt secret values in YAML files using SOPS (Mozilla). GitOps controller decrypts on apply.

# Encrypt specific fields in a Secret YAML:
sops --encrypt --in-place --encrypted-regex '^(data|stringData)$' secret.yaml

# Flux or ArgoCD configured with decryption provider
# On apply: decrypts in memory → creates K8s Secret
# Secret YAML committed to Git (encrypted values only)

HashiCorp Vault — Direct Integration

Vault offers the Vault Agent Injector: a mutating webhook that injects a Vault sidecar into Pods. The sidecar fetches secrets from Vault and writes them to a shared volume.

# Annotations trigger the injector:
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "web-app"
    vault.hashicorp.com/agent-inject-secret-db: "secret/data/db"
    vault.hashicorp.com/agent-inject-template-db: |
      {{- with secret "secret/data/db" -}}
      export DB_PASS={{ .Data.data.password }}
      {{- end -}}

4. Choosing the Right Approach

Decision Flowchart

Do you have an external secret store (Vault/AWS/GCP/Azure)?
├─ YES → Do you need secrets that never touch etcd?
│         ├─ YES → Secrets Store CSI Driver
│         └─ NO  → External Secrets Operator (simplest, most flexible)
└─ NO  → Do you use GitOps?
          ├─ YES → Sealed Secrets or SOPS
          └─ NO  → Native K8s Secrets + encryption at rest

5. Production Patterns

Secret Rotation Flow (ESO)

1. Secret rotated in AWS Secrets Manager (manual or Lambda-based auto-rotation)
2. ESO syncs new value on next refreshInterval
3. K8s Secret updated
4. If volume-mounted: kubelet syncs file in ~60s → app reads new value
5. If env var: kubectl rollout restart needed (or config hash annotation pattern)

Multi-Cluster Secret Sync

# Same ExternalSecret in all clusters pointing to same AWS path:
# dev cluster:  remoteRef.key = dev/database
# prod cluster: remoteRef.key = prod/database
# Single source of truth per environment
# Rotation in AWS propagates to all clusters automatically

Summary