Dynamic admission webhooks let you run your own code during the admission pipeline. The API server calls your webhook service (over HTTPS), sends the admission request, and your service responds with either an approval, a rejection, or a patch (mutation). This is how the entire policy and service mesh ecosystem works.

1. Webhook Architecture

The AdmissionReview Protocol

# API server sends (POST body):
{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "request": {
    "uid": "abc-123",
    "kind": {"group": "apps", "version": "v1", "kind": "Deployment"},
    "operation": "CREATE",
    "namespace": "production",
    "userInfo": {"username": "jane", "groups": ["developers"]},
    "object": { ... full Deployment spec ... },
    "oldObject": null  // populated for UPDATE operations
  }
}

# Webhook responds:
{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "abc-123",           // must match request UID
    "allowed": true,            // or false to reject
    "patchType": "JSONPatch",   // only for mutating webhooks
    "patch": "W3sib3AiOiJhZGQiLCJwYXRoIjoiL21ldGFkYXRhL2xhYmVscy90ZWFtIiwidmFsdWUiOiJwbGF0Zm9ybSJ9XQ=="
    // base64-encoded JSON Patch: [{"op":"add","path":"/metadata/labels/team","value":"platform"}]
  }
}

2. Webhook Configuration

ValidatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: policy-validator
webhooks:
  - name: validate.policy.example.com
    admissionReviewVersions: ["v1"]
    
    # WHAT to intercept:
    rules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        resources: ["deployments"]
        operations: ["CREATE", "UPDATE"]
        scope: Namespaced            # or Cluster, or "*"
    
    # WHERE to send:
    clientConfig:
      service:
        name: policy-webhook         # K8s Service (in-cluster)
        namespace: webhook-system
        path: /validate
        port: 443
      caBundle: LS0tLS1...           # CA cert to verify webhook's TLS
    
    # WHEN it fails:
    failurePolicy: Fail              # or Ignore
    
    # SCOPE control:
    namespaceSelector:               # Only intercept from matching namespaces
      matchExpressions:
        - key: environment
          operator: In
          values: ["production", "staging"]
    objectSelector:                  # Only intercept objects with matching labels
      matchLabels:
        policy-check: "true"
    
    # PERFORMANCE:
    timeoutSeconds: 5                # Max wait (default: 10s, max: 30s)
    sideEffects: None                # None, NoneOnDryRun (required for dry-run support)
    matchPolicy: Equivalent          # or Exact

Key Configuration Fields

failurePolicy — The Critical Decision

3. Mutating Webhooks

Mutating webhooks respond with a JSON Patch that modifies the object:

# MutatingWebhookConfiguration (similar structure to Validating):
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
  - name: inject.sidecar.example.com
    admissionReviewVersions: ["v1"]
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
        operations: ["CREATE"]
    clientConfig:
      service:
        name: sidecar-injector
        namespace: system
        path: /inject
      caBundle: LS0tLS1...
    failurePolicy: Ignore           # Don't block Pod creation if injector is down
    reinvocationPolicy: IfNeeded    # Re-call if another mutator changed the object
    namespaceSelector:
      matchLabels:
        sidecar-injection: enabled  # Only inject in labeled namespaces

Mutation Response (JSON Patch)

# Webhook returns patches like:
[
  {"op": "add", "path": "/spec/containers/-", "value": {
    "name": "envoy-sidecar",
    "image": "envoyproxy/envoy:v1.28",
    "ports": [{"containerPort": 15001}]
  }},
  {"op": "add", "path": "/metadata/labels/injected", "value": "true"},
  {"op": "add", "path": "/metadata/annotations/injector-version", "value": "v2.1"}
]
# Base64-encode this array → set as response.patch

reinvocationPolicy

4. TLS for Webhooks

Webhooks MUST serve HTTPS. The API server validates the webhook's TLS cert against the caBundle in the configuration.

Options for Managing Webhook TLS

# cert-manager annotation approach:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: policy-validator
  annotations:
    cert-manager.io/inject-ca-from: webhook-system/webhook-cert
# cert-manager automatically:
# 1. Issues a TLS cert for the webhook Service
# 2. Injects the CA bundle into the webhook configuration
# 3. Rotates before expiry

5. Operational Concerns

Avoiding Bootstrap Deadlocks

# Problem: Webhook in namespace "system" matches ALL Pod creates.
# If the webhook Pod crashes → no Pods can be created → webhook can't restart → DEADLOCK

# Solutions:
# 1. Exclude the webhook's own namespace:
namespaceSelector:
  matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values: ["webhook-system", "kube-system"]

# 2. Use objectSelector to skip specific Pods:
objectSelector:
  matchExpressions:
    - key: skip-webhook
      operator: DoesNotExist

# 3. Use failurePolicy: Ignore for non-critical webhooks

Performance Impact

# Every matching API request adds:
# - Network round-trip to webhook service (~1-5ms in-cluster)
# - Webhook processing time (depends on your code)
# - TLS handshake (amortized with connection pooling)

# Best practices:
# - Set timeoutSeconds low (3-5s) for latency-sensitive paths
# - Keep webhook logic fast (< 100ms)
# - Run multiple replicas (HA + distribute load)
# - Use namespaceSelector/objectSelector to narrow scope
# - Monitor: webhook_admission_duration_seconds metric

Debugging Webhook Issues

# Check if webhook is registered:
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# Check webhook Service/Endpoints:
kubectl get svc -n webhook-system
kubectl get endpoints -n webhook-system

# Check webhook Pod logs:
kubectl logs -n webhook-system -l app=webhook --tail=50

# Test with dry-run (triggers webhooks with sideEffects: None):
kubectl apply -f deploy.yaml --dry-run=server
# If rejected → webhook denied it

# Temporarily disable a broken webhook:
kubectl delete validatingwebhookconfiguration policy-validator
# ⚠️ Removes all enforcement — re-apply when fixed

Summary

[{"op":"add","path":"/spec/containers/-","value":{"name":"sidecar","image":"envoy:latest"}}]

namespaceSelector:
  matchExpressions:
    - key: environment
      operator: In
      values: ["production", "staging"]