Kubernetes Architecture Overview — Kubernetes Mastery

Why Kubernetes Exists

You already know containers. A single container runs one process in isolation. But in production, you have hundreds of containers across dozens of machines. Questions arise:

Which machine should this container run on?
What happens when a machine dies?
How do containers find and talk to each other?
How do you roll out a new version without downtime?

Kubernetes answers all of these. It's a declarative orchestration platform: you describe your desired state ("I want 3 replicas of this container, exposed on port 80"), and Kubernetes continuously works to make reality match that declaration.

Core Principle: Kubernetes is a reconciliation engine. You declare intent; controllers observe reality and act to close the gap. This "desired state → observe → diff → act" loop is the heartbeat of the entire system.

The Big Picture

A Kubernetes cluster has two planes:

Control Plane Components

The control plane makes global decisions. In production, it runs on dedicated machines (typically 3 or 5 for HA).

1. kube-apiserver — The Front Door

Every interaction with Kubernetes goes through the API server. kubectl, the scheduler, controllers, kubelets — they all talk to the API server via REST/gRPC. It is the only component that talks to etcd.

Responsibility	How
Authentication	Who are you? (certs, tokens, OIDC)
Authorization	Can you do this? (RBAC)
Admission Control	Should we allow/mutate this? (webhooks)
Validation	Is the object well-formed?
Persistence	Write to etcd
Watch notification	Notify watchers of changes

In managed clusters (EKS, GKE, AKS), you never see the API server process — the cloud provider runs it. But understanding its request pipeline is critical for debugging RBAC denials, admission webhook failures, and latency issues.

2. etcd — The Single Source of Truth

A distributed key-value store (based on the Raft consensus algorithm) that holds all cluster state. Every object you create — every Pod, Deployment, Secret — is a key in etcd.

Only the API server reads/writes etcd — no other component touches it directly
Uses a watch mechanism: the API server watches etcd for changes and fans them out
Stores data at paths like /registry/pods/default/my-pod

etcd is your most critical data store. Losing etcd without backups = losing your cluster state. Production clusters run 3 or 5 etcd nodes. Backup with etcdctl snapshot save regularly.

3. kube-scheduler — The Matchmaker

When a Pod is created but has no nodeName assigned, the scheduler picks the best node. Its algorithm:

Filter: Remove nodes that can't run the Pod (insufficient resources, taints, affinity rules)
Score: Rank remaining nodes (spread, resource balance, locality)
Bind: Assign the Pod to the highest-scoring node

Key: The scheduler doesn't move containers. It only makes a scheduling decision (writes nodeName to the Pod spec). The kubelet on that node then does the actual work of starting the container.

4. kube-controller-manager — The Reconciliation Loops

A single binary that bundles ~30 different controllers. Each controller watches a specific resource type and reconciles reality toward desired state:

Controller	Watches	Reconciles
ReplicaSet controller	ReplicaSets, Pods	Creates/deletes Pods to match `replicas`
Deployment controller	Deployments	Creates/updates ReplicaSets for rollouts
Node controller	Node heartbeats	Marks nodes as NotReady, evicts Pods
Job controller	Jobs, Pods	Runs Pods to completion
Endpoint controller	Services, Pods	Maintains endpoint lists for Services

Pattern: Every controller follows the same loop: Watch → Diff (desired vs actual) → Act. This is the most important architectural pattern in Kubernetes. When you write custom operators later, you'll implement this exact loop.

Worker Node Components

Worker nodes run your actual workloads. Each node has three components:

1. kubelet — The Node Agent

Runs on every node. Its job:

Watches the API server for Pods assigned to its node
Tells the container runtime to start/stop containers
Reports Pod and node status back to the API server
Runs liveness/readiness probes
Manages volumes (mount/unmount)

The kubelet does not run in a container — it's a systemd service on the host.

2. kube-proxy — The Network Plumber

Maintains network rules on the node so that Service ClusterIPs actually route to backing Pods. Traditionally uses iptables rules; modern clusters use IPVS or eBPF (Cilium).

3. Container Runtime

The software that pulls images and runs containers. Kubernetes talks to it via the Container Runtime Interface (CRI). Common runtimes: containerd, CRI-O.

How a Pod Gets Created — The Full Flow

Let's trace kubectl run nginx --image=nginx through the system:

Notice: No component tells another what to do. Instead, they all watch the API server and react to state changes. This is level-triggered (react to state) not edge-triggered (react to events). If a controller crashes and restarts, it simply re-reads current state and reconciles — no messages are lost.

Declarative vs Imperative

This is the fundamental mental model shift:

Imperative (traditional)	Declarative (Kubernetes)
"Start 3 nginx containers"	"I want 3 nginx replicas running"
You manage steps and failure recovery	K8s manages steps and failure recovery
If one dies, you detect and restart	Controller detects and restarts automatically
State is implicit (what's running?)	State is explicit (stored in etcd)

In practice, you almost always use declarative YAML manifests stored in Git (GitOps). kubectl apply -f is declarative; kubectl run is imperative. The CKA exam tests both, but production workflows are declarative.

Summary: The Mental Model

Memorize this sentence:

"The user declares desired state to the API server, which persists it in etcd. Controllers and the scheduler watch for changes, make decisions, and write back. The kubelet watches for Pods assigned to its node, and makes them real."

Every feature in Kubernetes — Deployments, Services, Ingress, RBAC, CRDs — is just another instance of this pattern: desired state in etcd → controller reconciles → reality converges.

What This Looks Like in Production

Concept	Production Reality
Control plane	3 API servers behind a load balancer, 3-5 etcd nodes (or managed by cloud provider)
Worker nodes	Autoscaling groups (10s to 1000s of nodes)
kubelet	Managed by systemd; if it dies, the node goes NotReady
etcd	Backed up hourly; size monitored (can hit quota limits)
API server	Rate-limited; audit-logged; fronted by admission webhooks for policy enforcement

📝 Quiz: Check Your Understanding

Q1: Which component is the ONLY one that directly reads/writes etcd?

The kube-apiserver. All other components interact with cluster state only through the API server.

Q2: A Pod has been created but is stuck in "Pending" status with no node assigned. Which component is responsible for the next step?

The kube-scheduler. It watches for Pods without a nodeName and assigns one. If it can't find a suitable node, the Pod stays Pending.

Q3: What's the difference between level-triggered and edge-triggered in the context of K8s controllers?

Level-triggered: react to the current state ("there are 2 Pods but I need 3 → create one"). Edge-triggered: react to a state transition ("a Pod just died → create one"). K8s is level-triggered — controllers reconcile based on current state, making them resilient to missed events.

Q4: If the kube-controller-manager crashes and restarts 5 minutes later, what happens to running Pods?

Nothing. Running Pods continue running — they're managed by kubelet. The controller manager simply re-reads current state from the API server and reconciles any drift it finds. No state is lost because truth lives in etcd, not in the controller's memory.

Q5: Why does Kubernetes use a separate scheduler instead of having the API server assign Pods to nodes directly?

Separation of concerns. The API server handles auth, validation, and persistence. The scheduler handles the complex optimization problem of placement. This separation means you can swap the scheduler (or run multiple schedulers) without changing the API server. It also means scheduling decisions don't block API requests.