In Lesson 1, you saw the big picture. Now we go deep into each control plane component — what it actually does, how it's configured, and what breaks in production.

1. kube-apiserver — The Heart of the Cluster

The API server is the only component that reads and writes to etcd. Every other component — scheduler, controllers, kubelet, even kubectl — talks exclusively to the API server. It is the cluster's central nervous system.

What It Actually Does

When a request arrives at the API server, it passes through a strict pipeline:

API Server as a Watch Hub

Beyond CRUD, the API server's most critical function is watch notifications. Controllers and kubelets open long-lived watch connections:

GET /api/v1/pods?watch=true&resourceVersion=12345

The API server streams change events (ADDED, MODIFIED, DELETED) for any objects matching the watch. This is how the entire system stays reactive without polling.

Running Multiple API Servers

In HA setups, you run 3+ API server instances behind a load balancer. They are stateless — all state lives in etcd. Any instance can serve any request. The load balancer just needs to health-check /healthz.

2. etcd — The Cluster's Memory

etcd is a distributed, consistent key-value store that uses the Raft consensus algorithm. It's where every Kubernetes object lives — Pods, Services, Secrets, ConfigMaps, everything.

How Data is Organized

/registry/pods/default/nginx-7d9fc44b6c-x2k4j
/registry/deployments/default/nginx
/registry/services/specs/default/kubernetes
/registry/secrets/kube-system/bootstrap-token-abcdef
/registry/nodes/worker-1

Each key stores the full serialized object (protobuf by default, JSON if configured).

Raft Consensus — Why 3 or 5 Nodes?

Raft requires a quorum (majority) to accept writes:

Key Operational Concepts

• resourceVersion: Every K8s object has one — it maps to etcd's ModRevision. Used for optimistic concurrency (conflict detection) and watch resumption.
• Compaction: etcd keeps history of all revisions. Compaction removes old revisions to reclaim space. K8s triggers this automatically.
• Defragmentation: After compaction, disk space isn't freed until defrag runs. Must be done per-member, sequentially.
• Quota: Default 2GB. If etcd exceeds quota, it goes read-only (alarm state). Your cluster effectively freezes for writes.

Backup and Restore

# Take a snapshot
etcdctl snapshot save /backup/etcd-$(date +%Y%m%d).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ca.crt \
  --cert=/etc/etcd/server.crt \
  --key=/etc/etcd/server.key

# Restore (stops the cluster!)
etcdctl snapshot restore /backup/etcd-20240101.db \
  --data-dir=/var/lib/etcd-restored

3. kube-scheduler — The Placement Engine

The scheduler's job is deceptively simple: for each unscheduled Pod, pick the best node. But "best" involves a complex scoring algorithm.

The Scheduling Cycle

Filter Plugins (Must Pass)

Score Plugins (Rank Nodes 0-100)

What Happens When No Node Fits?

The Pod stays Pending. The scheduler emits an event explaining why:

Events:
  Type     Reason            Message
  ----     ------            -------
  Warning  FailedScheduling  0/3 nodes are available:
                             1 Insufficient cpu,
                             2 node(s) had taint {key=dedicated:NoSchedule}

This is the first thing to check for stuck Pods: kubectl describe pod <name> and read the Events.

4. kube-controller-manager — The Reconciliation Engine

A single binary containing ~30 independent controllers, each running its own reconciliation loop. Think of it as 30 microservices compiled into one process for convenience.

The Controller Pattern

while true:
    desired = read_spec_from_api_server()
    actual  = observe_current_state()
    if actual != desired:
        take_action_to_converge(actual → desired)
    sleep(resync_interval)  # or react to watch events

Key Controllers and What They Do

Leader Election

Only one controller-manager instance is active at a time (the leader). Others are hot standbys. Leader election uses a Lease object in the kube-system namespace:

kubectl get lease -n kube-system kube-controller-manager
# holderIdentity: master-1_xxxx
# leaseDurationSeconds: 15
# renewTime: 2024-01-15T10:30:45Z

If the leader fails to renew within the lease duration, another instance takes over.

5. cloud-controller-manager — The Cloud Bridge

Introduced to decouple cloud-specific logic from the core Kubernetes codebase. It runs controllers that interact with your cloud provider's API:

In self-managed (bare-metal) clusters, there is no cloud-controller-manager. You either use MetalLB for LoadBalancer services or go without.

Control Plane Communication Map

Summary: What to Remember