This lesson ties together everything from Chapter 3 into a set of production-proven patterns. How should configuration flow from Git to running containers? How do you handle changes without downtime? How do you keep it maintainable across environments?
1. 12-Factor Config in Kubernetes
The 12-Factor App says: "Store config in the environment." Kubernetes gives you the tools to follow this rigorously:
| 12-Factor Principle | K8s Implementation |
|---|---|
| Config varies between deploys | ConfigMaps/Secrets per namespace (dev/staging/prod) |
| Strict separation of config from code | Config in ConfigMaps, not baked into images |
| Config injected via environment | Env vars or volume mounts at Pod start |
| No config in source code | Git stores references (ExternalSecret CRs), not values |
The Configuration Hierarchy
# Priority (highest → lowest): 1. Environment-specific overrides (ConfigMap per namespace) 2. Application defaults (baked into image as fallbacks) 3. Framework defaults (e.g., Spring Boot defaults) # Example: same app in dev and prod # dev namespace: ConfigMap with LOG_LEVEL=debug, REPLICAS_CACHE=1 # prod namespace: ConfigMap with LOG_LEVEL=warn, REPLICAS_CACHE=3 # Same image, different behavior
What Goes Where
| Data Type | Store In | Inject As |
|---|---|---|
| Feature flags, log levels | ConfigMap | Env vars or volume |
| Config files (nginx.conf, app.yaml) | ConfigMap | Volume mount |
| Database passwords, API keys | Secret (or External Secret) | Volume mount (preferred) |
| TLS certificates | Secret (type: kubernetes.io/tls) | Volume mount |
| Pod identity info | Downward API | Env vars |
| Service URLs | ConfigMap or K8s DNS | Env vars |
2. Hot-Reload Patterns
How do you update configuration without restarting Pods?
Pattern 1: Volume Mount Auto-Sync (Simplest)
# ConfigMap mounted as volume → files auto-update in ~60s # App watches file for changes (inotify, polling, or framework support) # Works for: nginx (with reload signal), Spring Boot (actuator refresh), # Prometheus (config reload endpoint)
# nginx reload on config change:
spec:
containers:
- name: nginx
image: nginx:1.25
volumeMounts:
- name: config
mountPath: /etc/nginx/conf.d
- name: reloader # ← sidecar watches for file changes
image: jimmidyson/configmap-reload:v0.9.0
args:
- --volume-dir=/etc/nginx/conf.d
- --webhook-url=http://localhost:80/-/reload
volumeMounts:
- name: config
mountPath: /etc/nginx/conf.d
Pattern 2: Reloader Operator (Stakater)
# Stakater Reloader watches ConfigMaps/Secrets and triggers
# rolling restarts of Deployments that reference them.
# Annotate the Deployment:
metadata:
annotations:
reloader.stakater.com/auto: "true"
# OR target specific ConfigMaps:
configmap.reloader.stakater.com/reload: "app-config"
# When app-config ConfigMap changes → Reloader triggers rollout restart
# No sidecar needed — cluster-level operator
Pattern 3: Config Hash Annotation (CI/CD Driven)
# CI/CD pipeline computes hash of ConfigMap content:
CONFIG_HASH=$(kubectl get configmap app-config -o json | sha256sum | cut -d' ' -f1)
# Patches the Deployment template annotation:
kubectl patch deployment web -p \
"{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"configHash\":\"$CONFIG_HASH\"}}}}}"
# Annotation change → template change → rolling update triggered
# Fully declarative in Helm/Kustomize:
spec:
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
Pattern 4: Immutable ConfigMaps + Name Versioning
# CI creates: app-config-v7 (immutable: true) # Deployment references: configMap.name: app-config-v7 # On config change: # 1. Create app-config-v8 # 2. Update Deployment to reference v8 # 3. Rolling update triggered naturally # 4. Rollback = point back to v7 # This is the safest pattern for critical systems
• No restart needed → Volume mount + app watches files (Pattern 1)
• Restart OK, want automation → Reloader operator (Pattern 2)
• Restart OK, want full control → Config hash in CI/CD (Pattern 3)
• Maximum safety → Immutable + versioned names (Pattern 4)
3. Multi-Environment Configuration
Kustomize Overlays (Built into kubectl)
# Directory structure:
base/
deployment.yaml
configmap.yaml
kustomization.yaml
overlays/
dev/
configmap-patch.yaml # LOG_LEVEL=debug
kustomization.yaml
prod/
configmap-patch.yaml # LOG_LEVEL=warn
kustomization.yaml
# Apply:
kubectl apply -k overlays/prod/
Helm Values Files
# values-dev.yaml: config: logLevel: debug replicas: 1 # values-prod.yaml: config: logLevel: warn replicas: 5 # helm install myapp ./chart -f values-prod.yaml
Namespace-per-Environment
# Simple and effective: # Namespace: dev → ConfigMap "app-config" (debug settings) # Namespace: prod → ConfigMap "app-config" (production settings) # Same Deployment manifest, different namespace = different config # Works because ConfigMaps are namespaced
base/ contains shared manifests, environments/{dev,staging,prod}/ contains Kustomize patches or Helm values files. ArgoCD/Flux applies the correct overlay per cluster. Configuration differences live in Git, not in someone's head.
4. Anti-Patterns to Avoid
| Anti-Pattern | Problem | Fix |
|---|---|---|
| Baking config into the image | Can't change without rebuild; different image per env | Externalize to ConfigMap/Secret |
| Storing Secrets in Git (even base64) | Trivially decoded; leaks on public repo | Use SealedSecrets, SOPS, or ESO |
| Env vars for large config files | Hard to read, no structure, line-length limits | Volume mount the config file |
| Single "mega-ConfigMap" | Any change triggers full resync; blast radius | Split by concern (app, logging, features) |
| No resource limits on config-heavy Pods | ConfigMap size (1MiB) can spike memory at mount | Always set resource requests/limits |
Using kubectl edit to change config | No audit trail, no Git history, not reproducible | All changes through Git + CI/CD |
| Env vars for secrets | Visible in /proc, logs, child processes | Volume mount with restrictive permissions |
5. Configuration Checklist
✅ Same image across all environments (config is external) ✅ Secrets in external store (Vault/AWS/GCP) — synced via ESO ✅ ConfigMaps split by concern (app, logging, features) ✅ Volume mounts for config files; env vars only for simple key-values ✅ Secrets mounted as files (not env vars) with 0400 permissions ✅ Immutable ConfigMaps in high-scale clusters ✅ Config hash annotation (or Reloader) for automatic rollouts ✅ All config changes go through Git (GitOps) ✅ readinessProbe on Pods that depend on config (fail fast on bad config) ✅ Encryption at rest enabled for Secrets in etcd
Summary
| Concept | Key Point |
|---|---|
| 12-Factor | One image, many environments. Config injected, not baked. |
| Hot-reload (no restart) | Volume mount + file watcher or sidecar reloader |
| Hot-reload (with restart) | Stakater Reloader, config hash annotation, or immutable + version |
| Multi-environment | Kustomize overlays or Helm values files per env |
| Safety | Immutable ConfigMaps + name versioning = safest pattern |
| Never | Bake config into image, store secrets in Git, use kubectl edit in prod |
📝 Quiz: Configuration Best Practices
Q1: Your app reads a config file at startup and never re-reads it. You update the ConfigMap. How do you apply the change?
kubectl rollout restart deployment/app, config hash annotation (triggers automatic rollout when config changes), or Stakater Reloader operator. Volume mount auto-sync won't help if the app only reads at startup.Q2: You have a Helm chart used in dev, staging, and prod. How do you manage per-environment configuration?
values-dev.yaml, values-staging.yaml, values-prod.yaml. Deploy with helm install -f values-prod.yaml. The chart template is the same; only values change. Store values files in Git alongside the chart.Q3: Your ConfigMap has 50 keys. One key changes frequently (feature flag), others are stable. What's the problem with one big ConfigMap?
app-config (stable) and feature-flags (volatile). Only the feature-flags change triggers re-sync.Q4: What does the Helm {{ sha256sum }} annotation pattern do?
Q5: Why is immutable ConfigMap + name versioning (app-config-v7) safer than mutable ConfigMaps?
(2) Clean rollback: Just change the reference back to v6 — no need to "undo" a ConfigMap edit.
(3) Audit trail: Each version is a separate object with its own creation timestamp.
(4) Performance: Immutable = no watches = less API server load.
Q6: An engineer uses kubectl edit configmap app-config to fix a production issue. Why is this an anti-pattern?