🗄️ Data Model — Keys, MVCC & Revisions

Key Namespace Structure

Kubernetes stores all objects under the /registry prefix. The path follows the pattern /registry/{group}/{resource}/{namespace}/{name} — or without namespace for cluster-scoped resources.

MVCC — Multi-Version Concurrency Control

etcd never overwrites values in place. Every write appends a new version keyed by the global revision (a monotonically increasing integer). Old versions are retained until compaction removes them. This is how the API server serves watch history without re-reading etcd.

# Every key has:
# - mod_revision:    etcd revision when last modified
# - create_revision: etcd revision when first created
# - version:         number of times this key has been modified

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
  get /registry/apps/deployments/default/nginx \
  --print-value-only | auger decode

# List all keys under a prefix
etcdctl get /registry/ --prefix --keys-only | head -30

# Get a key at a specific historical revision
etcdctl get /registry/apps/deployments/default/nginx \
  --rev=12345 --print-value-only | auger decode

# Watch for changes in real time
etcdctl watch /registry/apps/deployments/ --prefix

Revision vs resourceVersion

Transactions & Optimistic Locking

The API server uses etcd transactions with a compare-and-swap on mod_revision to prevent lost updates. If two clients try to update the same object concurrently, one gets a 409 Conflict — the loser must re-fetch and retry.

# etcd transaction: only update if revision matches (CAS)
etcdctl txn <<EOF
compares:
  mod("my-key") = "42"
success:
  put my-key new-value
failure:
  get my-key
EOF

🛠️ Operations — Backup, Compaction & Defrag

Snapshot Backup

etcd snapshots capture the entire keyspace at a point in time. Take them regularly — this is your only recovery path if the cluster state is lost.

# Take a snapshot (run from a control-plane node)
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key

# Verify the snapshot
etcdctl snapshot status /backup/etcd-20240101-020000.db --write-out=table
# +----------+----------+------------+------------+
# |   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
# +----------+----------+------------+------------+
# | abc12345 |   189423 |       3847 |     6.5 MB |
# +----------+----------+------------+------------+

Restore from Snapshot

# Stop kube-apiserver first (remove its static pod manifest or stop kubelet)
# Then restore on each member with a unique --name and --initial-advertise-peer-urls

ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-20240101-020000.db \
  --name=master-1 \
  --initial-cluster="master-1=https://10.0.0.1:2380,master-2=https://10.0.0.2:2380,master-3=https://10.0.0.3:2380" \
  --initial-cluster-token=etcd-cluster-1 \
  --initial-advertise-peer-urls=https://10.0.0.1:2380 \
  --data-dir=/var/lib/etcd-restore

# Update etcd's --data-dir to point to the restored directory, then restart

Compaction

Because MVCC retains all historical revisions, etcd grows unboundedly without compaction. Compaction discards all revisions older than a given revision number, freeing storage. The API server auto-compacts by default every 5 minutes.

# Get current revision
REV=$(etcdctl endpoint status --write-out=json | jq '.[0].Status.header.revision')

# Compact up to current revision (removes old MVCC versions)
etcdctl compact $REV

# Check remaining size
etcdctl endpoint status --write-out=table
# ENDPOINT              DB SIZE   ...
# https://127.0.0.1:2379  42 MB

Defragmentation

Compaction frees logical space but leaves fragmentation in the bbolt database file on disk. Defragmentation rewrites the file to reclaim physical disk space. It temporarily locks etcd — do it one member at a time.

# Defrag one member at a time (takes the member offline briefly)
etcdctl defrag --endpoints=https://10.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key

# Then defrag the next member
etcdctl defrag --endpoints=https://10.0.0.2:2379 ...

# Verify size reduced
etcdctl endpoint status --write-out=table

Health Checks

# Check health of all cluster members
etcdctl endpoint health --cluster \
  --endpoints=https://10.0.0.1:2379,https://10.0.0.2:2379,https://10.0.0.3:2379 \
  --cacert=... --cert=... --key=...

# Check who is the leader
etcdctl endpoint status --cluster --write-out=table
# ENDPOINT              ID          IS LEADER   IS LEARNER   RAFT TERM   ...
# https://10.0.0.1:2379 abc123      false        false        14          ...
# https://10.0.0.2:2379 def456      true         false        14          ...
# https://10.0.0.3:2379 ghi789      false        false        14          ...

# Check member list
etcdctl member list --write-out=table

📐 Sizing, Tuning & Production Tips

Hardware Recommendations

Key etcd Flags

# Election / heartbeat tuning
--heartbeat-interval=100       # ms between leader heartbeats (default 100)
--election-timeout=1000        # ms before follower starts election (default 1000)
# Rule: election-timeout should be 5-10x heartbeat-interval
# For high-latency networks (cross-region), increase both

# Storage
--quota-backend-bytes=8589934592   # 8 GB max DB size (default 2 GB)
--auto-compaction-mode=periodic
--auto-compaction-retention=1h     # compact revisions older than 1h

# Snapshot
--snapshot-count=10000             # entries between snapshots (default 10000)

# Security
--cert-file=/etc/etcd/etcd.crt
--key-file=/etc/etcd/etcd.key
--peer-cert-file=/etc/etcd/peer.crt
--peer-trusted-ca-file=/etc/etcd/ca.crt
--client-cert-auth=true
--peer-client-cert-auth=true

Production Best Practices

Critical etcd Metrics

📝 Knowledge Check