🛡️ Defense-in-Depth Security Model

Kubernetes security is layered — no single control is sufficient. Production clusters need independent controls at every layer so that a breach at one layer is contained by the next.

📋 CIS Benchmark & API Server Hardening

Run kube-bench

# Run CIS checks on a control-plane node
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-master.yaml
kubectl logs job/kube-bench-master

# Run on worker nodes
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-node.yaml
kubectl logs job/kube-bench-node

# Example output:
# [PASS] 1.2.1  Ensure --anonymous-auth is set to false
# [FAIL] 1.2.6  Ensure --kubelet-certificate-authority is set
# [WARN] 1.2.10 Ensure --tls-cipher-suites is set
# [INFO] 1.3.1  Ensure kube-controller-manager flags are set

Critical API Server Hardening Flags

Hardened Pod Security Context

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10000
        runAsGroup: 10000
        fsGroup: 10000
        seccompProfile:
          type: RuntimeDefault   # apply default seccomp filter
      containers:
        - name: app
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]         # drop every Linux capability
              add: ["NET_BIND_SERVICE"]  # add back only what's needed
          volumeMounts:
            - name: tmp
              mountPath: /tmp       # writable tmp for read-only root fs
      volumes:
        - name: tmp
          emptyDir: {}

🔍 Falco Runtime Security & Supply Chain

Install Falco

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=ebpf \        # prefer eBPF over kernel module
  --set falcosidekick.enabled=true \
  --set falcosidekick.config.slack.webhookurl=https://hooks.slack.com/... \
  --set falcosidekick.config.slack.minimumpriority=warning

Key Falco Rules (Built-in)

Custom Falco Rule

# Detect any curl/wget in a production container (data exfiltration)
- rule: Unexpected Network Tool in Container
  desc: Detects curl/wget execution in production pods
  condition: >
    spawned_process and container and
    proc.name in (curl, wget, nc, ncat, netcat) and
    not proc.pname in (package-manager, apt, yum)
  output: >
    Network tool executed in container
    (user=%user.name cmd=%proc.cmdline container=%container.name
     image=%container.image.repository:%container.image.tag)
  priority: WARNING
  tags: [network, exfiltration]

Image Supply Chain — Cosign Signing

# Sign image after CI build (keyless with OIDC)
cosign sign --yes ghcr.io/my-org/my-app:v1.2.3

# Verify signature
cosign verify \
  --certificate-identity=https://github.com/my-org/my-app/.github/workflows/build.yaml@refs/heads/main \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  ghcr.io/my-org/my-app:v1.2.3

Kyverno — Enforce Signed Images

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-image-signature
      match:
        resources: { kinds: [Pod] }
      verifyImages:
        - imageReferences: ["ghcr.io/my-org/*"]
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/my-org/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

🔑 Secrets Management & Audit Logs

Encrypt Secrets at Rest in etcd

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}   # fallback for unencrypted (migration)

# Pass to kube-apiserver:
# --encryption-provider-config=/etc/kubernetes/encryption-config.yaml

# Verify: check raw etcd value is encrypted
etcdctl get /registry/secrets/default/my-secret --print-value-only
# k8s:enc:aescbc:v1:key1:...   ← encrypted prefix confirms it worked

# Re-encrypt all existing secrets
kubectl get secrets -A -o json | kubectl replace -f -

External Secrets Operator (ESO)

Pull secrets from AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, or Azure Key Vault into Kubernetes Secrets — automatically rotated, never stored in Git.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: my-app
spec:
  refreshInterval: 1h         # re-sync every hour (picks up rotations)
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: db-credentials      # name of the K8s Secret to create
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: prod/my-app/db
        property: username
    - secretKey: password
      remoteRef:
        key: prod/my-app/db
        property: password

Audit Log Analysis for Security Events

# High-value audit log queries (using jq or your SIEM)

# Who accessed Secrets in the last hour?
cat /var/log/kubernetes/audit.log | jq -r '
  select(.objectRef.resource == "secrets" and .verb == "get") |
  [.requestReceivedTimestamp, .user.username, .objectRef.namespace, .objectRef.name]
  | @tsv' | sort | uniq

# Detect privilege escalation attempts
cat audit.log | jq -r '
  select(.responseStatus.code == 403) |
  [.user.username, .verb, .objectRef.resource, .requestReceivedTimestamp]
  | @tsv' | sort | uniq -c | sort -rn | head -20

# Who created ClusterRoleBindings? (RBAC escalation)
cat audit.log | jq -r '
  select(.objectRef.resource == "clusterrolebindings" and
         .verb in ["create","update","patch"]) |
  [.user.username, .objectRef.name, .requestReceivedTimestamp] | @tsv'

# kubectl exec into pods (lateral movement indicator)
cat audit.log | jq -r '
  select(.objectRef.subresource == "exec") |
  [.user.username, .objectRef.namespace, .objectRef.name, .requestReceivedTimestamp]
  | @tsv'

📝 Knowledge Check