🔄 The Full Request Pipeline

Every mutating or read request to the Kubernetes API server (whether from kubectl, a controller, or the Kubelet) passes through a deterministic chain of stages. Understanding this chain is essential for debugging, building admission webhooks, and designing secure multi-tenant clusters.

① AuthN ② AuthZ ③ Mutating Admission ④ Schema Validation ⑤ Validating Admission ⑥ Persist to etcd ⑦ Watch Propagation
1
Authentication (AuthN)

Who are you? TLS client cert, Bearer token (OIDC/ServiceAccount JWT), or webhook authenticator. Produces a UserInfo{username, groups, extra} object. Anonymous requests get system:anonymous.

2
Authorization (AuthZ)

Are you allowed to do this? RBAC, ABAC, Node, or Webhook authorizers checked in order. Any Allow wins; default is Deny. Checks verb + resource + namespace + name.

3
Mutating Admission

Webhooks may modify the object (inject sidecars, add labels, set defaults). Built-in plugins run first (e.g. DefaultStorageClass), then external MutatingWebhookConfigurations in order.

4
Schema / Object Validation

The (possibly mutated) object is validated against the resource's OpenAPI schema. CRD schemas use x-kubernetes-validations (CEL). Built-in types have hard-coded validation.

5
Validating Admission

Webhooks may reject the request (but cannot modify). OPA/Gatekeeper and Kyverno use this phase. All validating webhooks are called in parallel; any single rejection aborts the request.

6
Persist to etcd

Object is serialized (Protobuf by default), encoded, and written to etcd under the key /registry/<group>/<resource>/<namespace>/<name>. The resource version (resourceVersion) is the etcd revision number.

7
Watch Propagation

etcd streams a change event back to the API server's watch cache. All open Watch connections (controllers, Kubelet, kubectl get -w) receive the event. Controllers then enqueue a reconcile.

🔑 Stage 1 & 2 — Authentication & Authorization

Authentication Methods

MethodHow it worksCommon use
X.509 Client CertTLS mutual auth; CN = username, O = groupsNodes (system:node:…), admin kubeconfig
ServiceAccount JWTPod-mounted token at /var/run/secrets/…; signed by API serverAll in-cluster workloads
OIDC Bearer TokenJWT from external IdP (Dex, Okta, Keycloak); API server validates issuer + audienceHuman users, SSO
WebhookAPI server calls external HTTP endpoint with token; endpoint returns UserInfoCustom auth systems
Bootstrap TokenShort-lived token for node bootstrap (kubeadm join)New node registration
ℹ️ Projected Service Account Tokens Modern clusters use Bound ServiceAccount Tokens (projected volumes). They have a short TTL (default 1h), are audience-bound, and are automatically rotated by the Kubelet — unlike the old long-lived static tokens.

Authorization — RBAC Evaluation

After AuthN, the authorizer checks: "Is this user allowed to perform verb X on resource Y in namespace Z?" RBAC evaluates all matching RoleBindings and ClusterRoleBindings. The first Allow wins — there is no explicit Deny; missing rules default to Deny.

# Inspect what a ServiceAccount can do
kubectl auth can-i list pods \
  --as=system:serviceaccount:my-app:default \
  --namespace=my-app
# yes

# Dry-run a full permissions check
kubectl auth can-i --list \
  --as=system:serviceaccount:my-app:default \
  --namespace=my-app

# API server flag to enable all authorizers in order:
# --authorization-mode=Node,RBAC,Webhook

Mutating Admission — Built-in Plugins

Before external webhooks, built-in admission plugins run. Key ones:

PluginWhat it does
DefaultStorageClassSets storageClassName on PVCs with no class specified
DefaultTolerationSecondsAdds default node.kubernetes.io/not-ready tolerations
LimitRangerApplies default resource requests/limits from a LimitRange object
ServiceAccountInjects serviceAccountName and mounts ServiceAccount token
PodSecurityEnforces Pod Security Standards (restricted/baseline/privileged)
MutatingAdmissionWebhookCalls external MutatingWebhookConfiguration endpoints
⚠️ Webhook ordering matters MutatingWebhookConfigurations are called in alphabetical order by name. If webhook A sets a label that webhook B reads, name them accordingly (e.g. 00-injector, 01-validator).

✅ Stages 4 & 5 — Validation & Validating Admission

Schema Validation

After mutation, the API server validates the final object against the OpenAPI schema registered for that resource version. For CRDs this is the schema block in the CRD spec; for built-in types it's compiled Go validation.

# CEL validation in a CRD (Kubernetes 1.25+)
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
spec:
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 100
                maxUnavailable:
                  type: integer
              x-kubernetes-validations:
                # CEL rule: maxUnavailable must be <= replicas
                - rule: "self.maxUnavailable <= self.replicas"
                  message: "maxUnavailable cannot exceed replicas"
💡 Prefer CEL over webhooks for pure validation CEL rules run in-process — zero network round-trip, no latency, no webhook outage risk. Use ValidatingAdmissionWebhooks only when you need external data (e.g. database lookups, cross-resource checks).

Validating Admission Webhooks

All matching ValidatingWebhookConfigurations are called in parallel. Every one must return allowed: true; a single rejection (or timeout) aborts the request with the webhook's rejection message.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: policy-enforce
webhooks:
  - name: pods.policy.example.com
    admissionReviewVersions: ["v1"]
    clientConfig:
      service:
        name: policy-webhook
        namespace: policy-system
        path: /validate-pods
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    failurePolicy: Fail     # reject if webhook is unreachable
    timeoutSeconds: 5
    sideEffects: None
🚨 failurePolicy: Fail can block your cluster If a validating webhook is down and failurePolicy: Fail, ALL matching requests are rejected — including system components. Always set namespaceSelector to exclude kube-system and the webhook's own namespace.

Stage 6 — Persist to etcd

The validated, mutated object is serialized and written atomically to etcd using a Compare-And-Swap (CAS) on the resource version to prevent lost updates. The etcd revision becomes the new resourceVersion field.

# The etcd key path structure:
# /registry/{group}/{resource}/{namespace}/{name}
# Examples:
# /registry/apps/deployments/default/nginx
# /registry/core/pods/kube-system/coredns-abc123
# /registry/rbac.authorization.k8s.io/clusterroles/cluster-admin

# Inspect etcd directly (from a control-plane node):
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key \
  get /registry/apps/deployments/default/nginx --print-value-only \
  | auger decode    # auger decodes protobuf → YAML

Stage 7 — Watch Cache & Event Propagation

The API server maintains an in-memory watch cache that mirrors recent etcd history. Controllers connect with Watch requests and receive a stream of ADDED / MODIFIED / DELETED events without hitting etcd for every event.

Watch Cache

In-memory ring buffer of recent object versions. Serves LIST and WATCH from cache — reduces etcd load dramatically.

ResourceVersion

Clients pass their last-seen resourceVersion on reconnect. API server replays missed events from cache without etcd round-trips.

Informers

controller-runtime and client-go use Informers: a LIST+WATCH loop with a local store. Events flow to workqueues for reconcile.

Bookmark Events

Periodic synthetic events that advance the client's resourceVersion even when no real changes occur — prevents stale watches.

📋 Audit Logging & Extension Points

Audit Logging

Audit logging runs alongside the request pipeline — it records every API request at configurable verbosity levels. It is the primary forensic trail for incident investigation and compliance.

# Audit policy — controls what gets logged
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log nothing for read-only requests to non-sensitive resources
  - level: None
    verbs: ["get", "list", "watch"]
    resources:
      - group: ""
        resources: ["configmaps", "endpoints"]

  # Log metadata only for pod reads
  - level: Metadata
    verbs: ["get", "list", "watch"]
    resources:
      - group: ""
        resources: ["pods"]

  # Log full request+response for secrets
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]

  # Default: log metadata for everything else
  - level: Metadata
Audit LevelWhat's recorded
NoneNothing logged
MetadataRequest metadata (user, verb, resource, time) — no body
RequestMetadata + request body
RequestResponseMetadata + request body + response body

Key API Server Flags

# Authentication
--client-ca-file=/etc/kubernetes/pki/ca.crt
--service-account-issuer=https://kubernetes.default.svc
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key
--oidc-issuer-url=https://accounts.google.com
--oidc-client-id=my-app

# Authorization
--authorization-mode=Node,RBAC

# Admission
--enable-admission-plugins=NodeRestriction,PodSecurity,LimitRanger

# Audit
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100   # MB

Extension Points Summary

MutatingWebhook

Modify objects before validation. Used for: sidecar injection, label defaulting, image registry rewriting.

ValidatingWebhook

Reject objects that violate policy. Used by: OPA/Gatekeeper, Kyverno, custom policy engines.

CRD + CEL

Inline validation rules via Common Expression Language. No webhook needed — runs in-process at schema validation time.

AA (Aggregated API)

Register additional API groups served by an external API server (e.g. metrics-server exposes metrics.k8s.io).

📝 Knowledge Check

Q1. A kubectl apply of a new Pod is rejected with "admission webhook denied the request". At which stage in the pipeline did this happen?
  • A) Stage 1 — Authentication failed
  • B) Stage 2 — RBAC denied the verb
  • C) Stage 3 — A mutating webhook rejected it
  • D) Stage 5 — A validating admission webhook rejected it
D) Stage 5 — Validating Admission. Mutating webhooks (stage 3) can only modify objects, not reject them. Rejections come from validating webhooks (stage 5). The error message "admission webhook denied the request" is produced by ValidatingWebhookConfiguration handlers.
Q2. What does the resourceVersion field on a Kubernetes object represent?
  • A) The API version of the object's schema
  • B) A sequential counter incremented by the controller manager
  • C) The etcd revision number at the time the object was last written
  • D) A hash of the object's spec for detecting drift
C) The etcd revision number. Every write to etcd increments a global revision counter. The API server stores this as resourceVersion. It is used for optimistic concurrency (compare-and-swap) and as a bookmark for watch reconnects.
Q3. Your cluster has a ValidatingWebhookConfiguration with failurePolicy: Fail. The webhook pod crashes during a node outage. What is the immediate impact?
  • A) No impact — Kubernetes retries the webhook indefinitely
  • B) All matching API requests are rejected until the webhook recovers
  • C) The webhook is automatically bypassed after 30 seconds
  • D) Only CREATE requests are blocked; UPDATE and DELETE still work
B) All matching requests are rejected. With failurePolicy: Fail, if the webhook endpoint is unreachable (connection refused, timeout), the API server rejects the request as if the webhook returned denied. This can block control-plane operations. Mitigations: failurePolicy: Ignore for non-critical policies, namespace selectors to exclude system namespaces, and webhook HA (multiple replicas with PodDisruptionBudget).