This lesson ties together everything from Chapter 3 into a set of production-proven patterns. How should configuration flow from Git to running containers? How do you handle changes without downtime? How do you keep it maintainable across environments?

1. 12-Factor Config in Kubernetes

The 12-Factor App says: "Store config in the environment." Kubernetes gives you the tools to follow this rigorously:

12-Factor PrincipleK8s Implementation
Config varies between deploysConfigMaps/Secrets per namespace (dev/staging/prod)
Strict separation of config from codeConfig in ConfigMaps, not baked into images
Config injected via environmentEnv vars or volume mounts at Pod start
No config in source codeGit stores references (ExternalSecret CRs), not values

The Configuration Hierarchy

# Priority (highest → lowest):
1. Environment-specific overrides   (ConfigMap per namespace)
2. Application defaults              (baked into image as fallbacks)
3. Framework defaults                (e.g., Spring Boot defaults)

# Example: same app in dev and prod
# dev namespace:  ConfigMap with LOG_LEVEL=debug, REPLICAS_CACHE=1
# prod namespace: ConfigMap with LOG_LEVEL=warn,  REPLICAS_CACHE=3
# Same image, different behavior
The golden rule: one image, many environments. The same container image runs in dev, staging, and production. Only configuration changes. If you're building different images per environment, you're doing it wrong — you're not testing what you ship.

What Goes Where

Data TypeStore InInject As
Feature flags, log levelsConfigMapEnv vars or volume
Config files (nginx.conf, app.yaml)ConfigMapVolume mount
Database passwords, API keysSecret (or External Secret)Volume mount (preferred)
TLS certificatesSecret (type: kubernetes.io/tls)Volume mount
Pod identity infoDownward APIEnv vars
Service URLsConfigMap or K8s DNSEnv vars

2. Hot-Reload Patterns

How do you update configuration without restarting Pods?

Pattern 1: Volume Mount Auto-Sync (Simplest)

# ConfigMap mounted as volume → files auto-update in ~60s
# App watches file for changes (inotify, polling, or framework support)

# Works for: nginx (with reload signal), Spring Boot (actuator refresh),
#            Prometheus (config reload endpoint)
# nginx reload on config change:
spec:
  containers:
    - name: nginx
      image: nginx:1.25
      volumeMounts:
        - name: config
          mountPath: /etc/nginx/conf.d
    - name: reloader             # ← sidecar watches for file changes
      image: jimmidyson/configmap-reload:v0.9.0
      args:
        - --volume-dir=/etc/nginx/conf.d
        - --webhook-url=http://localhost:80/-/reload
      volumeMounts:
        - name: config
          mountPath: /etc/nginx/conf.d

Pattern 2: Reloader Operator (Stakater)

# Stakater Reloader watches ConfigMaps/Secrets and triggers
# rolling restarts of Deployments that reference them.

# Annotate the Deployment:
metadata:
  annotations:
    reloader.stakater.com/auto: "true"
# OR target specific ConfigMaps:
    configmap.reloader.stakater.com/reload: "app-config"

# When app-config ConfigMap changes → Reloader triggers rollout restart
# No sidecar needed — cluster-level operator

Pattern 3: Config Hash Annotation (CI/CD Driven)

# CI/CD pipeline computes hash of ConfigMap content:
CONFIG_HASH=$(kubectl get configmap app-config -o json | sha256sum | cut -d' ' -f1)

# Patches the Deployment template annotation:
kubectl patch deployment web -p \
  "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"configHash\":\"$CONFIG_HASH\"}}}}}"

# Annotation change → template change → rolling update triggered
# Fully declarative in Helm/Kustomize:
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

Pattern 4: Immutable ConfigMaps + Name Versioning

# CI creates: app-config-v7 (immutable: true)
# Deployment references: configMap.name: app-config-v7
# On config change:
#   1. Create app-config-v8
#   2. Update Deployment to reference v8
#   3. Rolling update triggered naturally
#   4. Rollback = point back to v7

# This is the safest pattern for critical systems
Choose based on your tolerance for Pod restarts:
• No restart needed → Volume mount + app watches files (Pattern 1)
• Restart OK, want automation → Reloader operator (Pattern 2)
• Restart OK, want full control → Config hash in CI/CD (Pattern 3)
• Maximum safety → Immutable + versioned names (Pattern 4)

3. Multi-Environment Configuration

Kustomize Overlays (Built into kubectl)

# Directory structure:
base/
  deployment.yaml
  configmap.yaml
  kustomization.yaml
overlays/
  dev/
    configmap-patch.yaml   # LOG_LEVEL=debug
    kustomization.yaml
  prod/
    configmap-patch.yaml   # LOG_LEVEL=warn
    kustomization.yaml

# Apply:
kubectl apply -k overlays/prod/

Helm Values Files

# values-dev.yaml:
config:
  logLevel: debug
  replicas: 1

# values-prod.yaml:
config:
  logLevel: warn
  replicas: 5

# helm install myapp ./chart -f values-prod.yaml

Namespace-per-Environment

# Simple and effective:
# Namespace: dev    → ConfigMap "app-config" (debug settings)
# Namespace: prod   → ConfigMap "app-config" (production settings)
# Same Deployment manifest, different namespace = different config
# Works because ConfigMaps are namespaced
In production GitOps repos, the common pattern is: base/ contains shared manifests, environments/{dev,staging,prod}/ contains Kustomize patches or Helm values files. ArgoCD/Flux applies the correct overlay per cluster. Configuration differences live in Git, not in someone's head.

4. Anti-Patterns to Avoid

Anti-PatternProblemFix
Baking config into the imageCan't change without rebuild; different image per envExternalize to ConfigMap/Secret
Storing Secrets in Git (even base64)Trivially decoded; leaks on public repoUse SealedSecrets, SOPS, or ESO
Env vars for large config filesHard to read, no structure, line-length limitsVolume mount the config file
Single "mega-ConfigMap"Any change triggers full resync; blast radiusSplit by concern (app, logging, features)
No resource limits on config-heavy PodsConfigMap size (1MiB) can spike memory at mountAlways set resource requests/limits
Using kubectl edit to change configNo audit trail, no Git history, not reproducibleAll changes through Git + CI/CD
Env vars for secretsVisible in /proc, logs, child processesVolume mount with restrictive permissions

5. Configuration Checklist

✅ Same image across all environments (config is external)
✅ Secrets in external store (Vault/AWS/GCP) — synced via ESO
✅ ConfigMaps split by concern (app, logging, features)
✅ Volume mounts for config files; env vars only for simple key-values
✅ Secrets mounted as files (not env vars) with 0400 permissions
✅ Immutable ConfigMaps in high-scale clusters
✅ Config hash annotation (or Reloader) for automatic rollouts
✅ All config changes go through Git (GitOps)
✅ readinessProbe on Pods that depend on config (fail fast on bad config)
✅ Encryption at rest enabled for Secrets in etcd

Summary

ConceptKey Point
12-FactorOne image, many environments. Config injected, not baked.
Hot-reload (no restart)Volume mount + file watcher or sidecar reloader
Hot-reload (with restart)Stakater Reloader, config hash annotation, or immutable + version
Multi-environmentKustomize overlays or Helm values files per env
SafetyImmutable ConfigMaps + name versioning = safest pattern
NeverBake config into image, store secrets in Git, use kubectl edit in prod

📝 Quiz: Configuration Best Practices

Q1: Your app reads a config file at startup and never re-reads it. You update the ConfigMap. How do you apply the change?

You must restart the Pods. Options: kubectl rollout restart deployment/app, config hash annotation (triggers automatic rollout when config changes), or Stakater Reloader operator. Volume mount auto-sync won't help if the app only reads at startup.

Q2: You have a Helm chart used in dev, staging, and prod. How do you manage per-environment configuration?

Use separate values files per environment: values-dev.yaml, values-staging.yaml, values-prod.yaml. Deploy with helm install -f values-prod.yaml. The chart template is the same; only values change. Store values files in Git alongside the chart.

Q3: Your ConfigMap has 50 keys. One key changes frequently (feature flag), others are stable. What's the problem with one big ConfigMap?

Blast radius. Any change to the ConfigMap causes kubelet to re-sync ALL mounted files (or triggers a full rollout if using config hash/Reloader). Split into separate ConfigMaps by concern: app-config (stable) and feature-flags (volatile). Only the feature-flags change triggers re-sync.

Q4: What does the Helm {{ sha256sum }} annotation pattern do?

It computes a SHA-256 hash of the ConfigMap template and puts it in the Pod template annotation. When the ConfigMap content changes, the hash changes → the Pod template changes → Kubernetes triggers a rolling update. This ensures Pods always restart when their config changes, without manual intervention.

Q5: Why is immutable ConfigMap + name versioning (app-config-v7) safer than mutable ConfigMaps?

(1) Atomic rollout: The Deployment references a specific version — all Pods get the same config (no race between old/new during sync).
(2) Clean rollback: Just change the reference back to v6 — no need to "undo" a ConfigMap edit.
(3) Audit trail: Each version is a separate object with its own creation timestamp.
(4) Performance: Immutable = no watches = less API server load.

Q6: An engineer uses kubectl edit configmap app-config to fix a production issue. Why is this an anti-pattern?

(1) No Git history — change isn't tracked, can't be code-reviewed. (2) Drift — next GitOps sync may overwrite it. (3) Not reproducible — can't recreate the same state from Git. (4) No rollback path — if the fix is wrong, no easy undo. Correct approach: commit the fix to Git, let CI/CD apply it.