A running container isn't necessarily a working container. Health checks let Docker know when your app is truly alive, and restart policies define what happens when it isn't. Together they form the foundation of self-healing infrastructure.

1. Why Health Checks?

Docker tracks whether a container's main process (PID 1) is running. But a process can be alive while the application inside is completely broken:

• Deadlock — threads are stuck waiting on each other; the process exists but serves nothing
• Memory exhaustion — the app is thrashing, responding to no requests
• Dependency failure — the database connection is gone; the app returns 500 on every request
• Startup hang — the process started but never finished initialization

Without health checks, Docker reports the container as "running" and load balancers keep sending traffic to a black hole. Health checks close this observability gap.

2. HEALTHCHECK Instruction

The HEALTHCHECK instruction in a Dockerfile tells Docker how to test whether a container is working:

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:3000/health || exit 1

Parameters Explained

Health States

A container with a health check moves through three states:

• starting — container just started; within the start-period grace window
• healthy — the health check command returned exit code 0
• unhealthy — the health check failed retries consecutive times

# Disable health check (e.g., in debug images)
HEALTHCHECK NONE

# Override at runtime
docker run --health-cmd="curl -f http://localhost/" \
  --health-interval=10s --health-retries=2 myapp

3. Health Check in Compose

In compose.yaml, health checks are defined with the healthcheck key:

services:
  api:
    image: myapp:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

  worker:
    image: myworker:latest
    depends_on:
      api:
        condition: service_healthy

The depends_on with condition: service_healthy ensures worker won't start until api passes its health check. This gives you proper startup ordering based on actual readiness, not just process start.

4. Custom Health Check Patterns

HTTP Endpoint (Most Common)

HEALTHCHECK CMD curl -f http://localhost:8080/health || exit 1

Best for web APIs. The /health endpoint should return 200 if the app is ready to serve traffic.

TCP Socket Check

HEALTHCHECK CMD nc -z localhost 5432 || exit 1

Good for databases and services where you just need to confirm the port is accepting connections.

Command Execution

HEALTHCHECK CMD redis-cli ping | grep -q PONG

Use the application's own CLI tool to verify internal state.

Database Connection Test

HEALTHCHECK CMD pg_isready -U postgres || exit 1

PostgreSQL ships pg_isready; MySQL has mysqladmin ping.

5. Restart Policies Revisited

Restart Policies + Health Checks: The Misconception

This means:

• Standalone Docker — health checks are informational only (visible in docker ps, events, inspect)
• Docker Swarm — the orchestrator kills and replaces unhealthy tasks
• Kubernetes — liveness probes trigger pod restarts
• Compose + autoheal — third-party tools like willfarrell/autoheal watch for unhealthy containers and restart them

6. Self-Healing Patterns

True self-healing requires three layers working together:

Layer 1: Application Resilience

The app itself handles transient failures — retries, circuit breakers, graceful degradation.

Layer 2: Container Restart

# Compose: restart on process crash
services:
  api:
    restart: on-failure:5
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 15s
      retries: 3

Layer 3: Orchestrator Recovery

# Kubernetes: liveness probe triggers pod restart
livenessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 10
  periodSeconds: 15
  failureThreshold: 3

7. Monitoring Health

Inspect Container Health

# Current health status
docker inspect --format='{{.State.Health.Status}}' mycontainer

# Full health details (last 5 check results)
docker inspect --format='{{json .State.Health}}' mycontainer | jq .

# Watch health events in real-time
docker events --filter event=health_status

Compose Health Overview

# See health status of all services
docker compose ps
# NAME    IMAGE    COMMAND    SERVICE    STATUS           PORTS
# api     myapp    ...        api        Up (healthy)     0.0.0.0:3000->3000/tcp
# db      pg:16    ...        db         Up (unhealthy)   5432/tcp

Integration with Monitoring

Health check results feed into monitoring stacks:

• Prometheus — export container health as a metric via cAdvisor
• Docker events — stream to Elasticsearch/Loki for alerting
• Uptime monitors — external tools (Pingdom, UptimeRobot) hit the same /health endpoint from outside

Hands-On Tasks

# Run nginx with a health check
docker run -d --name healthy-nginx \
  --health-cmd="curl -f http://localhost/ || exit 1" \
  --health-interval=5s \
  --health-retries=3 \
  --health-start-period=3s \
  nginx:alpine

# Watch it transition from 'starting' to 'healthy'
watch -n1 "docker inspect --format='{{.State.Health.Status}}' healthy-nginx"

# Now break it — block the port inside the container
docker exec healthy-nginx sh -c "nginx -s stop"

# Watch it go 'unhealthy' (process exits, container stops)
# Try with a subtler failure:
docker run -d --name broken-app \
  --health-cmd="curl -f http://localhost:8080/ || exit 1" \
  --health-interval=5s \
  --health-retries=2 \
  nginx:alpine
# nginx listens on 80, not 8080 — health check will fail
# Observe: container is "running" but "unhealthy"

docker inspect --format='{{json .State.Health}}' broken-app | jq .

# Cleanup
docker rm -f healthy-nginx broken-app

# compose.yaml
services:
  database:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: myapp
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 10s

  api:
    image: nginx:alpine  # stand-in for your app
    depends_on:
      database:
        condition: service_healthy
    ports:
      - "8080:80"

# Start and observe ordering
docker compose up

# You'll see:
# database: starting...
# database: healthy
# api: starting (only AFTER database is healthy)

# Verify
docker compose ps  # both show "(healthy)" or "Up"
docker compose down

Interactive Quizzes

Key Takeaways

• Process alive ≠ app healthy — always define health checks for production containers
• HEALTHCHECK gives Docker three states: starting → healthy ↔ unhealthy
• Keep checks fast — under 100ms, testing only critical-path readiness
• Restart policies react to process exit, NOT health status — don't confuse the two
• Orchestrators close the gap — Swarm/K8s kill and replace unhealthy containers
• Compose ordering — use depends_on: condition: service_healthy for true startup sequencing
• Kubernetes three-probe pattern — startup, liveness, readiness each serve a different purpose
• Self-healing is layered — app resilience + restart policies + orchestrator recovery