By default, Kubernetes stores all data in etcd as plain text (base64 for Secrets is just encoding, not security). Encryption at rest ensures that even if an attacker gains access to etcd's data files, they cannot read sensitive information. This is a CKS exam staple and a production security requirement.

1. The Threat Model

What encryption at rest protects against:

2. EncryptionConfiguration

The API server reads an EncryptionConfiguration file that defines which resources to encrypt and which encryption provider to use.

# /etc/kubernetes/enc/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets                  # Encrypt Secrets
      - configmaps               # Can also encrypt ConfigMaps
    providers:                   # Tried in order for WRITES; all tried for READS
      - aescbc:                  # ← Primary: used for all new writes
          keys:
            - name: key-2024-01
              secret: dGhpcy1pcy1hLTMyLWJ5dGUta2V5LTEyMzQ1Ng==  # 32-byte base64
      - identity: {}             # ← Fallback: can read unencrypted (old) data

Provider Ordering Rules

• First provider = used for writing (encrypting new/updated resources)
• All providers = tried in order for reading (decrypting existing resources)
• identity: {} = reads unencrypted data (needed during migration from plaintext to encrypted)

3. Encryption Providers

Local Key Providers (aescbc, secretbox)

# Generate a 32-byte key:
head -c 32 /dev/urandom | base64
# Output: dGhpcy1pcy1hLTMyLWJ5dGUta2V5LTEyMzQ1Ng==

# Use in config:
providers:
  - secretbox:
      keys:
        - name: key-2024-01
          secret: dGhpcy1pcy1hLTMyLWJ5dGUta2V5LTEyMzQ1Ng==

KMS v2 Envelope Encryption

Envelope encryption adds a layer of indirection:

# KMS v2 provider configuration:
providers:
  - kms:
      apiVersion: v2
      name: my-kms-provider
      endpoint: unix:///var/run/kms-plugin/socket.sock
      timeout: 3s

4. Step-by-Step: Enable Encryption at Rest

Step 1: Generate Key and Create Config

# Generate a 32-byte encryption key:
ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)

# Create the config file:
cat > /etc/kubernetes/enc/encryption-config.yaml <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key-$(date +%Y%m%d)
              secret: ${ENCRYPTION_KEY}
      - identity: {}
EOF

# Set restrictive permissions:
chmod 600 /etc/kubernetes/enc/encryption-config.yaml

Step 2: Configure the API Server

# Edit /etc/kubernetes/manifests/kube-apiserver.yaml:
spec:
  containers:
    - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
        # ... other flags
      volumeMounts:
        - name: enc
          mountPath: /etc/kubernetes/enc
          readOnly: true
  volumes:
    - name: enc
      hostPath:
        path: /etc/kubernetes/enc
        type: DirectoryOrCreate

The API server static Pod restarts automatically when its manifest changes.

Step 3: Re-encrypt Existing Secrets

# Wait for API server to restart, then:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

# This reads each Secret (decrypts with old provider or identity)
# and writes it back (encrypts with new primary provider)

Step 4: Verify

# Read directly from etcd (bypass API server):
ETCDCTL_API=3 etcdctl get /registry/secrets/default/db-creds \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | hexdump -C

# Should see: k8s:enc:aescbc:v1:key-20240115:... (encrypted bytes)
# NOT: {"apiVersion":"v1","data":... (plaintext JSON)

5. Key Rotation

Encryption keys should be rotated periodically. The process for local keys:

Rotation Steps

1. Add new key as first in the list (it becomes the write key)
2. Keep old key(s) below it (still used for reading old data)
3. Restart API server
4. Re-encrypt all Secrets (they get re-encrypted with new key)
5. Remove old key from config (no data is encrypted with it anymore)
6. Restart API server again

# After rotation, the config looks like:
providers:
  - aescbc:
      keys:
        - name: key-2024-06    # ← NEW key (writes)
          secret: bmV3LWtleS0yMDI0LTA2LXJvdGF0aW9u...
        - name: key-2024-01    # ← OLD key (reads only, until re-encryption done)
          secret: dGhpcy1pcy1hLTMyLWJ5dGUta2V5LTEyMzQ1Ng==
  - identity: {}

Removing identity (Disabling Unencrypted Reads)

Once all Secrets are encrypted and verified, you can remove identity: {} from the providers list. This ensures the API server cannot read unencrypted data — any plaintext Secret left behind will cause a read error (which alerts you to the problem).

Summary