🔄 The Full Request Pipeline
Every mutating or read request to the Kubernetes API server (whether from kubectl, a controller, or the Kubelet) passes through a deterministic chain of stages. Understanding this chain is essential for debugging, building admission webhooks, and designing secure multi-tenant clusters.
Who are you? TLS client cert, Bearer token (OIDC/ServiceAccount JWT), or webhook authenticator. Produces a UserInfo{username, groups, extra} object. Anonymous requests get system:anonymous.
Are you allowed to do this? RBAC, ABAC, Node, or Webhook authorizers checked in order. Any Allow wins; default is Deny. Checks verb + resource + namespace + name.
Webhooks may modify the object (inject sidecars, add labels, set defaults). Built-in plugins run first (e.g. DefaultStorageClass), then external MutatingWebhookConfigurations in order.
The (possibly mutated) object is validated against the resource's OpenAPI schema. CRD schemas use x-kubernetes-validations (CEL). Built-in types have hard-coded validation.
Webhooks may reject the request (but cannot modify). OPA/Gatekeeper and Kyverno use this phase. All validating webhooks are called in parallel; any single rejection aborts the request.
Object is serialized (Protobuf by default), encoded, and written to etcd under the key /registry/<group>/<resource>/<namespace>/<name>. The resource version (resourceVersion) is the etcd revision number.
etcd streams a change event back to the API server's watch cache. All open Watch connections (controllers, Kubelet, kubectl get -w) receive the event. Controllers then enqueue a reconcile.
🔑 Stage 1 & 2 — Authentication & Authorization
Authentication Methods
| Method | How it works | Common use |
|---|---|---|
| X.509 Client Cert | TLS mutual auth; CN = username, O = groups | Nodes (system:node:…), admin kubeconfig |
| ServiceAccount JWT | Pod-mounted token at /var/run/secrets/…; signed by API server | All in-cluster workloads |
| OIDC Bearer Token | JWT from external IdP (Dex, Okta, Keycloak); API server validates issuer + audience | Human users, SSO |
| Webhook | API server calls external HTTP endpoint with token; endpoint returns UserInfo | Custom auth systems |
| Bootstrap Token | Short-lived token for node bootstrap (kubeadm join) | New node registration |
Authorization — RBAC Evaluation
After AuthN, the authorizer checks: "Is this user allowed to perform verb X on resource Y in namespace Z?" RBAC evaluates all matching RoleBindings and ClusterRoleBindings. The first Allow wins — there is no explicit Deny; missing rules default to Deny.
# Inspect what a ServiceAccount can do
kubectl auth can-i list pods \
--as=system:serviceaccount:my-app:default \
--namespace=my-app
# yes
# Dry-run a full permissions check
kubectl auth can-i --list \
--as=system:serviceaccount:my-app:default \
--namespace=my-app
# API server flag to enable all authorizers in order:
# --authorization-mode=Node,RBAC,Webhook
Mutating Admission — Built-in Plugins
Before external webhooks, built-in admission plugins run. Key ones:
| Plugin | What it does |
|---|---|
DefaultStorageClass | Sets storageClassName on PVCs with no class specified |
DefaultTolerationSeconds | Adds default node.kubernetes.io/not-ready tolerations |
LimitRanger | Applies default resource requests/limits from a LimitRange object |
ServiceAccount | Injects serviceAccountName and mounts ServiceAccount token |
PodSecurity | Enforces Pod Security Standards (restricted/baseline/privileged) |
MutatingAdmissionWebhook | Calls external MutatingWebhookConfiguration endpoints |
00-injector, 01-validator).
✅ Stages 4 & 5 — Validation & Validating Admission
Schema Validation
After mutation, the API server validates the final object against the OpenAPI schema registered for that resource version. For CRDs this is the schema block in the CRD spec; for built-in types it's compiled Go validation.
# CEL validation in a CRD (Kubernetes 1.25+)
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
spec:
versions:
- name: v1
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
replicas:
type: integer
minimum: 1
maximum: 100
maxUnavailable:
type: integer
x-kubernetes-validations:
# CEL rule: maxUnavailable must be <= replicas
- rule: "self.maxUnavailable <= self.replicas"
message: "maxUnavailable cannot exceed replicas"
Validating Admission Webhooks
All matching ValidatingWebhookConfigurations are called in parallel. Every one must return allowed: true; a single rejection (or timeout) aborts the request with the webhook's rejection message.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: policy-enforce
webhooks:
- name: pods.policy.example.com
admissionReviewVersions: ["v1"]
clientConfig:
service:
name: policy-webhook
namespace: policy-system
path: /validate-pods
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
failurePolicy: Fail # reject if webhook is unreachable
timeoutSeconds: 5
sideEffects: None
failurePolicy: Fail, ALL matching requests are rejected — including system components. Always set namespaceSelector to exclude kube-system and the webhook's own namespace.
Stage 6 — Persist to etcd
The validated, mutated object is serialized and written atomically to etcd using a Compare-And-Swap (CAS) on the resource version to prevent lost updates. The etcd revision becomes the new resourceVersion field.
# The etcd key path structure:
# /registry/{group}/{resource}/{namespace}/{name}
# Examples:
# /registry/apps/deployments/default/nginx
# /registry/core/pods/kube-system/coredns-abc123
# /registry/rbac.authorization.k8s.io/clusterroles/cluster-admin
# Inspect etcd directly (from a control-plane node):
ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/peer.crt \
--key=/etc/kubernetes/pki/etcd/peer.key \
get /registry/apps/deployments/default/nginx --print-value-only \
| auger decode # auger decodes protobuf → YAML
Stage 7 — Watch Cache & Event Propagation
The API server maintains an in-memory watch cache that mirrors recent etcd history. Controllers connect with Watch requests and receive a stream of ADDED / MODIFIED / DELETED events without hitting etcd for every event.
Watch Cache
In-memory ring buffer of recent object versions. Serves LIST and WATCH from cache — reduces etcd load dramatically.
ResourceVersion
Clients pass their last-seen resourceVersion on reconnect. API server replays missed events from cache without etcd round-trips.
Informers
controller-runtime and client-go use Informers: a LIST+WATCH loop with a local store. Events flow to workqueues for reconcile.
Bookmark Events
Periodic synthetic events that advance the client's resourceVersion even when no real changes occur — prevents stale watches.
📋 Audit Logging & Extension Points
Audit Logging
Audit logging runs alongside the request pipeline — it records every API request at configurable verbosity levels. It is the primary forensic trail for incident investigation and compliance.
# Audit policy — controls what gets logged
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log nothing for read-only requests to non-sensitive resources
- level: None
verbs: ["get", "list", "watch"]
resources:
- group: ""
resources: ["configmaps", "endpoints"]
# Log metadata only for pod reads
- level: Metadata
verbs: ["get", "list", "watch"]
resources:
- group: ""
resources: ["pods"]
# Log full request+response for secrets
- level: RequestResponse
resources:
- group: ""
resources: ["secrets"]
# Default: log metadata for everything else
- level: Metadata
| Audit Level | What's recorded |
|---|---|
None | Nothing logged |
Metadata | Request metadata (user, verb, resource, time) — no body |
Request | Metadata + request body |
RequestResponse | Metadata + request body + response body |
Key API Server Flags
# Authentication
--client-ca-file=/etc/kubernetes/pki/ca.crt
--service-account-issuer=https://kubernetes.default.svc
--service-account-signing-key-file=/etc/kubernetes/pki/sa.key
--oidc-issuer-url=https://accounts.google.com
--oidc-client-id=my-app
# Authorization
--authorization-mode=Node,RBAC
# Admission
--enable-admission-plugins=NodeRestriction,PodSecurity,LimitRanger
# Audit
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100 # MB
Extension Points Summary
MutatingWebhook
Modify objects before validation. Used for: sidecar injection, label defaulting, image registry rewriting.
ValidatingWebhook
Reject objects that violate policy. Used by: OPA/Gatekeeper, Kyverno, custom policy engines.
CRD + CEL
Inline validation rules via Common Expression Language. No webhook needed — runs in-process at schema validation time.
AA (Aggregated API)
Register additional API groups served by an external API server (e.g. metrics-server exposes metrics.k8s.io).
📝 Knowledge Check
kubectl apply of a new Pod is rejected with "admission webhook denied the request". At which stage in the pipeline did this happen?resourceVersion field on a Kubernetes object represent?resourceVersion. It is used for optimistic concurrency (compare-and-swap) and as a bookmark for watch reconnects.failurePolicy: Fail. The webhook pod crashes during a node outage. What is the immediate impact?failurePolicy: Fail, if the webhook endpoint is unreachable (connection refused, timeout), the API server rejects the request as if the webhook returned denied. This can block control-plane operations. Mitigations: failurePolicy: Ignore for non-critical policies, namespace selectors to exclude system namespaces, and webhook HA (multiple replicas with PodDisruptionBudget).