🏗️ Kubelet Architecture

The kubelet is the node agent — a long-running process on every node (including control-plane nodes). It watches the API server for Pods assigned to its node (spec.nodeName == thisNode), then drives each Pod through its full lifecycle using three plugin interfaces: CRI (container runtime), CNI (networking), and CSI (storage).

🚀 Pod Lifecycle — Admit to Running

1 — Admission

When the kubelet sees a new Pod assigned to its node it first runs admission handlers — these are local checks (not API server admission):

• Resource availability — enough CPU/memory on node, not exceeding node allocatable
• Node selectors / affinity — should have been enforced by scheduler but kubelet double-checks
• Eviction thresholds — if the node is already under memory pressure, the pod may be rejected

2 — Image Pull

Kubelet asks the CRI (containerd) to pull required images. Pull policy controls this:

3 — Sandbox & Volume Setup

Kubelet creates the pause container (infra/sandbox) first. This container holds the network namespace shared by all containers in the pod. Then CNI is called to attach the network interface and assign the pod IP. Then CSI volumes are mounted.

4 — Init Containers

Init containers run sequentially to completion before any app containers start. A failure restarts the init container (per restartPolicy). Use cases: DB schema migration, config file generation, waiting for dependencies.

spec:
  initContainers:
    - name: wait-for-db
      image: busybox
      command: ['sh', '-c',
        'until nc -z postgres-svc 5432; do echo waiting; sleep 2; done']
    - name: run-migrations
      image: myapp:latest
      command: ['./migrate', '--up']
  containers:
    - name: app
      image: myapp:latest

5 — Lifecycle Hooks & Probes

spec:
  containers:
    - name: app
      lifecycle:
        postStart:               # runs immediately after container starts
          exec:
            command: ["/bin/sh", "-c", "echo started > /tmp/ready"]
        preStop:                 # runs before SIGTERM — use for graceful drain
          exec:
            command: ["/bin/sh", "-c", "nginx -s quit; sleep 5"]
      startupProbe:              # delays liveness until app is ready to start
        httpGet: { path: /healthz, port: 8080 }
        failureThreshold: 30
        periodSeconds: 10
      livenessProbe:             # restart container if unhealthy
        httpGet: { path: /healthz, port: 8080 }
        initialDelaySeconds: 0
        periodSeconds: 10
        failureThreshold: 3
      readinessProbe:            # remove from Service endpoints if not ready
        httpGet: { path: /ready, port: 8080 }
        periodSeconds: 5
        failureThreshold: 3

🔌 CRI, CNI & CSI Integration

CRI — Container Runtime Interface

The kubelet talks to the container runtime via a gRPC API defined by CRI. This decouples the kubelet from any specific runtime. The two main CRI implementations are containerd and CRI-O.

# Inspect containers via crictl (bypasses kubelet/docker)
crictl ps                          # running containers
crictl pods                        # running sandboxes
crictl images                      # cached images
crictl logs <container-id>         # container stdout/stderr
crictl exec -it <container-id> sh  # shell into container
crictl inspect <container-id>      # full container spec + state

# Check CRI socket kubelet is using
ps aux | grep kubelet | grep container-runtime-endpoint
# or in kubelet config: --container-runtime-endpoint=unix:///run/containerd/containerd.sock

CNI — Container Network Interface

After the CRI creates the pod sandbox, the kubelet invokes the CNI plugin binary to configure networking. CNI plugins are executables in /opt/cni/bin/ called with a JSON config from /etc/cni/net.d/.

# CNI is called with ADD/DEL/CHECK commands
# ADD: called when sandbox is created — assigns IP, sets up veth pair, routes
# DEL: called when sandbox is deleted — tears down networking
# CHECK: verifies network state matches config

# CNI config example (Flannel)
cat /etc/cni/net.d/10-flannel.conflist
# {
#   "name": "cbr0",
#   "plugins": [
#     { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } },
#     { "type": "portmap", "capabilities": { "portMappings": true } }
#   ]
# }

# View pod network namespace
POD_ID=$(crictl pods --name myapp -q)
crictl inspectp $POD_ID | jq '.status.network'

CSI — Container Storage Interface

The kubelet's Volume Manager interacts with CSI drivers via gRPC to attach, mount, and unmount volumes. The flow for a PVC-backed volume:

1. Attach — CSI controller plugin attaches the cloud disk to the node (EBS AttachVolume)
2. Mount — CSI node plugin mounts the device to a global mount point on the node
3. Bind-mount — Kubelet bind-mounts from the global path into the pod's container filesystem

# Check volume attachment status
kubectl get volumeattachments
# NAME                               ATTACHER                  PV           NODE     ATTACHED
# csi-abc123...                      ebs.csi.aws.com           pvc-xyz...   node-1   true

# Inspect mounted volumes on a node
ls /var/lib/kubelet/pods/<pod-uid>/volumes/
# kubernetes.io~configmap/  kubernetes.io~secret/  kubernetes.io~csi/

# Check if a CSI driver is registered
kubectl get csidrivers
kubectl get csinodes

📊 Node Status, Eviction & Production Tips

Node Status Reporting

Kubelet updates the Node object in the API server every --node-status-update-frequency (default 10s). It reports conditions, allocatable resources, images cached, and attached volumes.

# Node conditions reported by kubelet
kubectl describe node worker-1 | grep -A 20 Conditions
# Type                 Status   Reason
# MemoryPressure       False    KubeletHasSufficientMemory
# DiskPressure         False    KubeletHasNoDiskPressure
# PIDPressure          False    KubeletHasSufficientPID
# Ready                True     KubeletReady

# Allocatable vs Capacity — kubelet reserves resources for OS + kubelet itself
kubectl get node worker-1 -o json | jq '{capacity: .status.capacity, allocatable: .status.allocatable}'
# capacity.memory:    16Gi
# allocatable.memory: 15.5Gi  ← ~500Mi reserved for system

Node-Pressure Eviction

When the eviction manager detects resource pressure it evicts pods — lowest priority and highest consumption first — to protect node stability.

# Configure eviction thresholds in kubelet config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
  memory.available: "200Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
evictionSoft:
  memory.available: "500Mi"
  nodefs.available: "15%"
evictionSoftGracePeriod:
  memory.available: "1m30s"
  nodefs.available: "2m"
evictionMaxPodGracePeriod: 90   # cap on graceful termination during eviction

Key kubelet Configuration Flags

# Preferred: use KubeletConfiguration file
--config=/etc/kubernetes/kubelet-config.yaml

# Key settings:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd                    # must match container runtime (containerd)
maxPods: 110                             # max pods per node (default 110)
kubeReserved:                            # reserve for kubelet/OS
  cpu: "200m"
  memory: "512Mi"
systemReserved:
  cpu: "200m"
  memory: "256Mi"
nodeStatusUpdateFrequency: "10s"
imageGCHighThresholdPercent: 85          # start GC when image fs > 85%
imageGCLowThresholdPercent: 80           # stop GC when image fs < 80%
containerLogMaxSize: "50Mi"              # per-container log rotation size
containerLogMaxFiles: 5

📝 Knowledge Check