Containers unlock elastic scaling — but scaling well requires understanding the patterns, trade-offs, and deployment strategies that keep services reliable under changing load.

1. Horizontal vs Vertical Scaling

Vertical scaling means giving a single container more resources — more CPU cores, more memory. You're making a bigger box.

Horizontal scaling means running more containers (replicas) behind a load balancer. You're adding more boxes.

Containers are designed for horizontal scaling:

• Stateless — each replica is identical, no shared local state
• Identical — built from the same image, guaranteed consistency
• Fast to start — seconds, not minutes like VMs
• Cheap — minimal overhead per instance

2. Manual Scaling

When you know the load pattern (e.g., marketing campaign at 9 AM), you can scale manually:

# Docker Swarm
docker service scale web=5

# Kubernetes
kubectl scale deployment web --replicas=5

# Docker Compose (v2)
docker compose up --scale web=5

Manual scaling is appropriate when:

• Load is predictable and scheduled
• You want human oversight before adding capacity
• Cost control is critical (autoscalers can overshoot)

3. Autoscaling

Autoscaling adjusts replica count based on real-time metrics:

Kubernetes HPA (Horizontal Pod Autoscaler)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Target-based scaling: "Keep average CPU at 70%" — the HPA calculates how many pods are needed: desired = ceil(current * (currentMetric / targetMetric)).

AWS Auto Scaling

ECS and Fargate support target tracking policies with the same concept — pick a metric, set a target, let the platform adjust task count.

4. Scale to Zero

Serverless containers take horizontal scaling to its logical endpoint: if there's no traffic, run zero replicas.

Cold start trade-off: Going from 0→1 takes time (pull image, start process, warm up). Mitigations: keep minimum=1, use small images, pre-warm on schedule.

When to use scale-to-zero:

• Event-driven workloads (process a webhook, then idle)
• Batch/cron jobs
• Low-traffic services (dev/staging, internal tools)
• Cost-sensitive environments

5. Deployment Strategies

How you update services at scale determines downtime, risk, and rollback speed.

Strategy Details

# Kubernetes rolling update (default)
kubectl set image deployment/web web=myapp:v2

# Control the rollout speed
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # 1 extra pod during update
      maxUnavailable: 0  # never go below desired count

6. Scaling Challenges

Stateful Services

Databases, caches with local state, and file-based systems can't simply add replicas. Solutions:

• Use managed services (RDS, Cloud SQL) — let the provider handle replication
• StatefulSets in K8s with persistent volumes
• Read replicas for read-heavy workloads

Session Affinity

If your app stores sessions in memory, requests from the same user must hit the same pod. Better fix: externalize sessions to Redis/Memcached.

Connection Limits

Database connection pools don't scale linearly. 50 pods × 10 connections = 500 connections to your DB. Use connection poolers (PgBouncer, ProxySQL).

Thundering Herd

When autoscaler adds 10 pods simultaneously, they all start hitting the same cold caches, warming up at once. Mitigations: stagger startup, pre-warm caches, rate-limit on cold start.

Graceful Scale-Down

When removing replicas, you must drain existing connections:

# Kubernetes handles this with:
# 1. Pod removed from Service endpoints (no new traffic)
# 2. SIGTERM sent to container
# 3. preStop hook + terminationGracePeriodSeconds for draining
spec:
  terminationGracePeriodSeconds: 30
  containers:
  - lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]

7. Capacity Planning

Resource Requests and Limits (Kubernetes)

resources:
  requests:          # Guaranteed minimum (used for scheduling)
    cpu: 250m        # 0.25 CPU cores
    memory: 256Mi
  limits:            # Hard ceiling (throttled/OOM-killed)
    cpu: 1000m
    memory: 512Mi

Bin-Packing Efficiency

Schedulers fit pods onto nodes like items in a bin. Over-requesting wastes capacity; under-requesting causes contention.

Right-Sizing Containers

Monitoring informs scaling: Use metrics (Prometheus, Datadog) to observe actual resource usage and tune requests/limits. Tools like Goldilocks and VPA (Vertical Pod Autoscaler) can recommend values.

Interactive Quizzes

Hands-On Task

# 1. Initialize Swarm (if not already)
docker swarm init

# 2. Create a service with v1
docker service create \
  --name web \
  --replicas 2 \
  --publish 8080:80 \
  --update-delay 5s \
  --update-parallelism 1 \
  nginx:1.24-alpine

# 3. Verify it's running
docker service ps web
curl http://localhost:8080

# 4. Scale to 5 replicas
docker service scale web=5

# 5. Watch replicas come up
docker service ps web
# All 5 should show "Running"

# 6. In another terminal, run a continuous health check
while true; do
  curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080
  sleep 0.5
done

# 7. Perform a rolling update to v2
docker service update \
  --image nginx:1.25-alpine \
  --update-parallelism 1 \
  --update-delay 5s \
  web

# 8. Watch the rolling update progress
docker service ps web
# You'll see old tasks shutting down, new ones starting

# 9. Verify zero downtime
# The health check loop should show uninterrupted 200s

# 10. Rollback if needed
docker service rollback web

# Cleanup
docker service rm web

Industry Callout

Key Takeaways

• Horizontal over vertical — containers are built for horizontal scaling; keep them stateless and identical
• Autoscale on the right metric — CPU isn't always the answer; use the metric that represents actual demand (queue depth, RPS, custom)
• Scale to zero saves money — but cold starts have a cost; balance latency vs. spend
• Choose your deployment strategy — rolling for simplicity, blue-green for instant rollback, canary for lowest risk
• Stateful services need special care — externalize state, use connection poolers, plan for graceful shutdown
• Right-size your containers — monitor actual usage, set appropriate requests/limits, avoid waste and OOMs
• Scaling is a system problem — consider databases, caches, DNS, and downstream services, not just your app