The control plane makes decisions. Worker nodes execute them. Every container you run lives on a worker node, managed by three components working in concert: the kubelet, kube-proxy, and the container runtime.

1. kubelet — The Node Agent

The kubelet is a systemd service (not a container) running on every worker node. It is the bridge between the Kubernetes API and the container runtime on that machine.

What the kubelet Does

1. Watches the API server for Pods assigned to its node (via spec.nodeName)
2. Translates Pod specs into container runtime calls (via CRI)
3. Manages Pod lifecycle — starts, stops, restarts containers
4. Runs probes — liveness, readiness, startup
5. Reports status — Pod conditions and node conditions back to the API server
6. Manages volumes — mounts/unmounts as Pods start/stop
7. Enforces resources — sets cgroup limits for CPU/memory
8. Evicts Pods — when node is under resource pressure

PLEG — Pod Lifecycle Event Generator

The kubelet doesn't poll the container runtime constantly. Instead, PLEG periodically (every 1 second) queries the runtime for the list of all containers and compares it to the last known state. Changes generate events that trigger the kubelet's sync loop.

# A common production issue:
PLEG is not healthy
# This means PLEG took >3min to relist containers.
# Causes: overloaded node, slow container runtime, too many containers on one node.

kubelet Pod Admission

Before starting a Pod, the kubelet runs its own admission checks (separate from API server admission):

• Resource check: Does the node have enough allocatable CPU/memory?
• PID pressure: Is the node running out of process IDs?
• Node affinity: Double-check (in case of race conditions)
• Tolerations: Can the Pod tolerate the node's taints?

If admission fails, the Pod stays Pending with a reason like OutOfCpu or OutOfMemory.

Node Registration

When the kubelet starts, it self-registers with the API server — creating or updating its Node object with:

• Capacity (total CPU, memory, max pods)
• Allocatable (capacity minus system-reserved and kube-reserved)
• Node conditions (Ready, MemoryPressure, DiskPressure, PIDPressure)
• Node labels and annotations

# Check allocatable vs capacity:
kubectl describe node worker-1 | grep -A5 "Allocatable"
# Allocatable:
#   cpu:     3800m       (4 cores - 200m reserved)
#   memory:  7400Mi      (8Gi - 600Mi reserved)
#   pods:    110         (default max)

2. kube-proxy — Service Networking

kube-proxy makes Service objects work. When you create a Service with a ClusterIP, kube-proxy ensures that any traffic sent to that virtual IP gets routed to one of the backing Pods.

Three Modes

How iptables Mode Works

For each Service, kube-proxy creates iptables rules that:

1. DNAT packets destined for the ClusterIP to a randomly-selected Pod IP
2. Apply probability-based load balancing (each Pod gets 1/N chance)
3. Handle session affinity if configured

# Example: Service "web" (10.96.0.100:80) → 3 Pods
# kube-proxy generates rules like:
-A KUBE-SERVICES -d 10.96.0.100/32 -p tcp --dport 80 -j KUBE-SVC-XXXXX
-A KUBE-SVC-XXXXX -m statistic --mode random --probability 0.333 -j KUBE-SEP-AAA
-A KUBE-SVC-XXXXX -m statistic --mode random --probability 0.500 -j KUBE-SEP-BBB
-A KUBE-SVC-XXXXX -j KUBE-SEP-CCC
-A KUBE-SEP-AAA -p tcp -j DNAT --to-destination 10.244.1.5:8080
-A KUBE-SEP-BBB -p tcp -j DNAT --to-destination 10.244.2.3:8080
-A KUBE-SEP-CCC -p tcp -j DNAT --to-destination 10.244.3.7:8080

Why IPVS at Scale?

iptables rules are O(n) — every packet traverses the chain linearly. With 10,000 Services × 10 endpoints = 100,000 rules, this gets slow. IPVS uses hash tables (O(1) lookup) and supports multiple load-balancing algorithms: round-robin, least-connections, weighted, etc.

The "No kube-proxy" Option

Modern CNI plugins like Cilium can replace kube-proxy entirely using eBPF. Instead of iptables/IPVS rules, Cilium programs eBPF maps in the kernel — faster lookups, more features (topology-aware routing, Maglev hashing), and easier debugging.

3. Container Runtime & CRI

The kubelet doesn't run containers directly. It speaks the Container Runtime Interface (CRI) — a gRPC API — to a container runtime that does the actual work.

CRI Architecture

CRI API — Two Services

containerd vs CRI-O

4. Pod Sandbox Lifecycle

A Pod is not just "a group of containers." At the OS level, a Pod is a sandbox — a set of shared Linux namespaces that containers join.

The Pause Container

Every Pod has a hidden "pause" container (also called the "infra container"). It's a tiny process that:

1. Creates and holds the Pod's network namespace
2. Acts as PID 1 (reaps zombie processes)
3. Stays running for the Pod's entire lifetime

All other containers in the Pod join the pause container's namespaces. This is why containers in the same Pod share:

Full Pod Startup Sequence

Pod Shutdown Sequence

When a Pod is deleted, the kubelet executes a graceful shutdown:

1. Pod enters Terminating state
2. Pod is removed from Service endpoints (stops receiving traffic)
3. preStop hook runs (if defined)
4. SIGTERM sent to PID 1 in each container
5. Wait up to terminationGracePeriodSeconds (default: 30s)
6. If still running: SIGKILL
7. CNI: detach network
8. Remove sandbox

5. Eviction Manager — When the Node Runs Out

The kubelet monitors node resources and evicts Pods when thresholds are breached:

Eviction Order (QoS-based)

1. BestEffort Pods (no requests/limits) — evicted first
2. Burstable Pods exceeding their requests
3. Guaranteed Pods (requests = limits) — evicted last, only in extreme pressure

Node Status — What the Control Plane Sees

kubectl describe node worker-1

Conditions:
  Type             Status  Reason
  ----             ------  ------
  Ready            True    KubeletReady
  MemoryPressure   False   KubeletHasSufficientMemory
  DiskPressure     False   KubeletHasNoDiskPressure
  PIDPressure      False   KubeletHasSufficientPID

# If Ready becomes Unknown (kubelet stops reporting):
# → Node controller waits 40s, then marks NotReady
# → After pod-eviction-timeout (5min), Pods are evicted

Summary