A container isn't just "running" or "stopped." It's a state machine with discrete states, well-defined transitions, and kernel-level mechanisms behind each one. Understanding this machine is essential to designing reliable architectures that handle crashes, restarts, and graceful shutdowns.
1. The State Machine
Every Docker container passes through a set of discrete states. Each transition is triggered by a specific command or system event:
- Created — Container exists (writable layer + config allocated) but no process running
- Running — PID 1 process is active inside the container's namespaces
- Paused — Process frozen via cgroup freezer; still in memory, consuming no CPU
- Stopped (Exited) — PID 1 has terminated; writable layer and metadata still on disk
- Dead — Partially removed container in a broken state (resource cleanup failed)
2. Create vs Run
docker run is actually two operations combined: docker create + docker start.
What docker create does
- Pulls the image (if not cached)
- Allocates a writable container layer (copy-on-write filesystem)
- Stores the container configuration (env vars, mounts, network, entrypoint)
- Assigns a container ID — returns it without starting any process
When to use create separately
- Init containers — pre-create containers that run setup tasks before the main app
- Pre-warming — create containers ahead of time so startup latency is just the process fork, not image pull + layer allocation
- Batch scheduling — create a pool of containers, then start them in a controlled sequence
- Configuration injection — inspect/modify the config between create and start
# Create without starting
docker create --name my-app -p 8080:80 nginx:alpine
# Container is now in "Created" state
docker inspect --format '{{.State.Status}}' my-app
# → "created"
# Start when ready
docker start my-app
# → "running"
3. Starting & Stopping
Starting a container
When you issue docker start, the Docker daemon:
- Sets up namespaces (PID, NET, MNT, UTS, IPC, USER)
- Configures cgroups for resource limits
- Prepares the overlay filesystem mount
- Forks the container's PID 1 process (the entrypoint)
- If the container was previously stopped, sends SIGCONT to resume
Stopping a container
docker stop follows a two-phase shutdown:
# Stop with default 10s grace period
docker stop my-app
# Stop with custom grace period (30 seconds for DB to drain)
docker stop --time 30 my-app
# Immediate kill (no grace period)
docker kill my-app
If your application doesn't handle SIGTERM, Docker will SIGKILL it after the grace period. This means:
- Database connections are severed mid-transaction — potential data corruption
- In-flight HTTP requests get dropped — clients see errors
- Write buffers aren't flushed — log data or queued events are lost
- File locks aren't released — next container start may fail
Always trap SIGTERM in your application and shut down cleanly.
4. Pause & Unpause
docker pause uses the cgroup freezer subsystem to suspend all processes in the container. Unlike stop, the processes remain in memory — they're just not scheduled by the CPU.
| Aspect | Pause | Stop |
|---|---|---|
| Mechanism | cgroup freezer (SIGSTOP-like) | SIGTERM → SIGKILL |
| Process state | Frozen in memory | Terminated |
| Memory usage | Retained | Released |
| Resume speed | Instant | Full process startup |
| Network | Connections preserved (may timeout) | Connections closed |
Use cases for pause
- Filesystem snapshots — freeze the container to get a consistent snapshot of the writable layer (
docker commit) - Temporary resource relief — free CPU without the cost of full restart
- Debugging — freeze a misbehaving container to inspect its state without it continuing to cause damage
# Pause a running container
docker pause my-app
docker inspect --format '{{.State.Status}}' my-app
# → "paused"
# Resume it
docker unpause my-app
5. Restart Policies
Restart policies tell the Docker daemon what to do when a container exits. They are set at container creation time with --restart:
| Policy | Behavior | Use Case |
|---|---|---|
no |
Never restart (default) | One-off tasks, batch jobs, debugging |
on-failure[:max] |
Restart only if exit code ≠ 0, up to max times | Workers that should retry but not loop forever |
always |
Restart regardless of exit code; also restarts on daemon startup | Long-running services that must always be up |
unless-stopped |
Like always, but doesn't restart if explicitly stopped before daemon restart |
Services you want auto-recovery but manual control over |
The Docker daemon distinguishes between a container that crashes (exit code ≠ 0) and one that was explicitly stopped (docker stop). With unless-stopped, an explicit stop is respected — the daemon won't restart it. With always, even an explicitly stopped container restarts when the daemon itself restarts.
# Restart on failure, max 5 attempts
docker run -d --restart=on-failure:5 --name worker my-worker-image
# Always restart (survives daemon restarts)
docker run -d --restart=always --name api my-api-image
# Check restart count
docker inspect --format '{{.RestartCount}}' worker
6. Removing Containers
Stopped containers still consume disk space (writable layer + metadata). You must explicitly remove them:
# Remove a stopped container
docker rm my-app
# Force-remove a running container (sends SIGKILL first)
docker rm -f my-app
# Remove all stopped containers
docker container prune
# Auto-remove when container exits (great for one-off tasks)
docker run --rm alpine echo "I clean up after myself"
Orphaned containers and disk space
Every stopped container retains its writable layer. On a busy CI server running hundreds of builds, this accumulates fast:
# See how much space stopped containers use
docker system df
# Typical output on an unmanaged host:
# Containers: 847 Size: 12.4GB (reclaimable: 11.8GB)
--rm for One-Off TasksAny container you run for a quick test, build step, or data migration should use --rm. This automatically removes the container (and its writable layer) the moment it exits. You avoid accumulating ghost containers and wasting disk.
7. Exit Codes
When a container stops, its exit code tells you why. Learn to read them:
| Exit Code | Meaning | Common Cause |
|---|---|---|
0 |
Success | Process completed normally |
1 |
Application error | Unhandled exception, config error, missing dependency |
137 |
SIGKILL (128 + 9) | OOM killer, docker kill, exceeded memory limit |
143 |
SIGTERM (128 + 15) | docker stop (app handled SIGTERM and exited) |
126 |
Command not executable | Permission denied on entrypoint |
127 |
Command not found | Entrypoint binary doesn't exist in the image |
# Check exit code of a stopped container
docker inspect --format '{{.State.ExitCode}}' my-app
# Debugging workflow
docker ps -a --filter "exited=137" # Find OOM-killed containers
docker logs my-app # Check app logs before death
docker inspect my-app | jq '.[0].State' # Full state details
If you see exit code 137 and you didn't manually docker kill, your container hit its memory limit. The kernel's OOM killer terminated it. Fix: increase the memory limit (--memory) or fix the memory leak in your application. Check docker inspect for OOMKilled: true.
Interactive Quizzes
Quiz 1: State Transitions
A container is in the "Running" state. You issue docker pause followed by docker unpause then docker stop. What is the final state?
Quiz 2: Exit Codes
Your container exited with code 137. You did NOT manually run docker kill. What most likely happened?
docker kill manually, the most common cause is the kernel OOM killer terminating the process for exceeding its memory limit.Quiz 3: Restart Policies
A container is running with --restart=unless-stopped. You run docker stop on it, then restart the Docker daemon. What happens to the container?
unless-stopped, if you explicitly stop a container, the daemon remembers that decision and won't restart it on daemon restart. This is the key difference from always, which restarts the container regardless.Hands-On Tasks
🛠️ Task 1: Walk the State Machine
Create a container, inspect each state, and observe the exit code:
# Step 1: Create (don't start)
docker create --name lifecycle-test alpine sleep 30
docker inspect --format '{{.State.Status}}' lifecycle-test
# Expected: "created"
# Step 2: Start it
docker start lifecycle-test
docker inspect --format '{{.State.Status}}' lifecycle-test
# Expected: "running"
# Step 3: Pause and inspect
docker pause lifecycle-test
docker inspect --format '{{.State.Status}}' lifecycle-test
# Expected: "paused"
# Step 4: Unpause, then stop
docker unpause lifecycle-test
docker stop --time 5 lifecycle-test
docker inspect --format '{{.State.Status}}' lifecycle-test
# Expected: "exited"
# Step 5: Check exit code
docker inspect --format '{{.State.ExitCode}}' lifecycle-test
# Expected: 137 (SIGKILL after grace period, since sleep doesn't handle SIGTERM)
# Clean up
docker rm lifecycle-test
🛠️ Task 2: Test Restart Policies
Run a container that crashes, and observe Docker's restart behavior:
# Run a container that exits with error code 1 after 2 seconds
docker run -d --name restart-test \
--restart=on-failure:3 \
alpine sh -c "echo 'Starting...'; sleep 2; echo 'Crashing!'; exit 1"
# Watch the restart count increase (check every 5 seconds)
watch -n 5 'docker inspect --format "Restarts: {{.RestartCount}} | Status: {{.State.Status}}" restart-test'
# After 3 restarts, the container stays stopped
# You should eventually see: Restarts: 3 | Status: exited
# Inspect the full restart history
docker inspect --format '{{json .RestartCount}}' restart-test
# Compare with a successful exit (no restart triggered)
docker run -d --name success-test \
--restart=on-failure:3 \
alpine sh -c "echo 'Done'; exit 0"
docker inspect --format '{{.RestartCount}}' success-test
# Expected: 0
# Clean up
docker rm -f restart-test success-test
Kubernetes builds heavily on container lifecycle primitives but adds orchestration-level hooks:
- preStop hook — a command or HTTP call executed before SIGTERM is sent. Use it to deregister from service discovery, drain connections, or finish processing.
- terminationGracePeriodSeconds — Kubernetes' equivalent of
docker stop --time. Defaults to 30s. After this, the kubelet sends SIGKILL. Set it high enough for your app's drain logic. - restartPolicy — Kubernetes Pods have
Always(default for Deployments),OnFailure(Jobs), andNever. The kubelet handles restarts with exponential backoff (10s, 20s, 40s… up to 5 minutes). - Container states in Pods — Kubernetes tracks Waiting, Running, and Terminated for each container, with reason codes like
OOMKilled,Error,Completed.
# Kubernetes pod spec with lifecycle hooks
spec:
terminationGracePeriodSeconds: 60
containers:
- name: api
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5 && kill -TERM 1"]
Key Takeaways
- Containers are state machines — Created → Running → Paused/Stopped → Dead. Know what triggers each transition.
docker run= create + start — separate them when you need pre-warming or staged orchestration.- Always handle SIGTERM — your app gets 10 seconds by default to shut down gracefully. Use them to close connections, flush writes, and deregister from load balancers.
- Pause ≠ Stop — pause freezes the process in place (cgroup freezer); stop terminates it. Choose based on whether you need the memory back.
- Restart policies are your first line of resilience — use
on-failurefor workers,unless-stoppedfor services, neveralwaysfor one-off tasks. - Exit codes tell the story — 137 means OOM/kill, 143 means graceful SIGTERM, 127 means missing binary. Read them before reading logs.
- Clean up after yourself — use
--rmfor ephemeral containers,docker container pruneregularly, and monitordocker system df.