Production Cluster Setup — Kubernetes Mastery

🏗️ HA Control-Plane Architecture

A production Kubernetes cluster requires a highly available control plane — at minimum 3 control-plane nodes behind a load balancer, each running kube-apiserver, kube-controller-manager, kube-scheduler, and etcd (or pointing to an external etcd ring).

3 Control-Plane Nodes

Minimum for HA — tolerates 1 node failure. Use 5 nodes for critical clusters that must tolerate 2 simultaneous failures.

Load Balancer for API

All clients (kubectl, kubelets, controllers) point to a VIP or cloud NLB. Never point directly at a control-plane IP.

Co-located vs External etcd

Co-located: simpler, fewer nodes. External: etcd on dedicated nodes for better isolation and independent scaling.

Multi-AZ Placement

Spread control-plane and etcd nodes across 3 availability zones. Losing one AZ must not lose quorum.

Co-located vs External etcd

Topology	Pros	Cons	Use when
Co-located (stacked)	Fewer nodes, simpler ops	Control-plane and etcd share fate; harder to scale etcd independently	Most clusters — recommended default
External etcd	Independent scaling; etcd failure doesn't kill control plane	More nodes to manage (3 CP + 3 etcd = 6 nodes minimum)	Very large clusters; strict compliance requirements

📐 Node Sizing & Cluster Topology

🟢 Small (< 50 nodes)

3× control plane: 4 vCPU, 8 GB
Workers: 4–8 vCPU, 16–32 GB
etcd: co-located on CP nodes
kube-proxy: iptables OK
CNI: Flannel or Calico
Storage: cloud CSI driver

🟡 Medium (50–500 nodes)

3× control plane: 8 vCPU, 16 GB
Workers: 8–16 vCPU, 32–64 GB
etcd: co-located or external
kube-proxy: IPVS mode
CNI: Cilium or Calico
Storage: CSI + default StorageClass

🔴 Large (500+ nodes)

5× control plane: 16 vCPU, 32 GB
Workers: varies by workload
etcd: dedicated 5-node ring (NVMe)
kube-proxy: eBPF / Cilium
CNI: Cilium with WireGuard encryption
Storage: Rook/Ceph or cloud

Worker Node Sizing Principles

Principle	Recommendation
Node count vs size	Fewer large nodes = less overhead; more nodes = better blast radius isolation. Balance at ~32–64 vCPU per node.
Max pods per node	Default 110; limited by IP space (CNI subnet size) and kernel limits. Increase with care — more pods = heavier kubelet.
CPU overcommit	2–4× overcommit is normal for mixed workloads; 1× for latency-sensitive apps. Monitor actual utilisation.
Memory overcommit	1–1.5× max. Memory overcommit causes OOM kills — more dangerous than CPU throttling.
Dedicated node pools	Separate node pools for: system (kube-system), GPU, spot/preemptible, batch, and production workloads.

CNI Plugin Selection

CNI	Routing model	Network policy	eBPF	Best for
Flannel	VXLAN overlay	No (needs Calico)	No	Simple dev/test clusters
Calico	BGP or VXLAN	Yes (NetworkPolicy + CiliumNetworkPolicy)	Partial	On-prem, BGP peering with routers
Cilium	eBPF + VXLAN/BGP	Yes (L3–L7)	Full	Cloud-native, high scale, security
AWS VPC CNI	Native VPC IPs	Via security groups	No	EKS — no overlay overhead
GKE Dataplane V2	eBPF (Cilium)	Yes	Full	GKE managed clusters

💡 Choose CNI before cluster creation Changing CNI after cluster creation requires draining all nodes and restarting all pods — it's essentially a cluster rebuild. Make the right choice upfront.

Default StorageClass

# Ensure exactly one default StorageClass — multiple defaults cause PVC binding failures
kubectl get sc
# NAME                        PROVISIONER             DEFAULT
# gp3 (default)               ebs.csi.aws.com         ✓
# gp2                         ebs.csi.aws.com

# Set a StorageClass as default
kubectl patch storageclass gp3 \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# Unset old default
kubectl patch storageclass gp2 \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

🚀 HA Bootstrap with kubeadm

kubeadm Config File

# kubeadm-config.yaml — first control-plane node
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.29.0
controlPlaneEndpoint: "k8s-api.example.com:6443"  # LB VIP or DNS
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
  dnsDomain: "cluster.local"
etcd:
  local:
    dataDir: /var/lib/etcd
    extraArgs:
      auto-compaction-mode: periodic
      auto-compaction-retention: "1h"
      quota-backend-bytes: "8589934592"  # 8 GB
apiServer:
  extraArgs:
    audit-log-path: /var/log/kubernetes/audit.log
    audit-policy-file: /etc/kubernetes/audit-policy.yaml
    enable-admission-plugins: NodeRestriction,PodSecurity
    oidc-issuer-url: https://accounts.google.com
    oidc-client-id: kubernetes
  certSANs:
    - k8s-api.example.com
    - 10.0.0.10             # LB internal IP
controllerManager:
  extraArgs:
    bind-address: "0.0.0.0"   # expose metrics to Prometheus
scheduler:
  extraArgs:
    bind-address: "0.0.0.0"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "10.0.0.1"  # this node's IP
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  kubeletExtraArgs:
    cloud-provider: external
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
serverTLSBootstrap: true
rotateCertificates: true
maxPods: 110
kubeReserved:
  cpu: "250m"
  memory: "512Mi"
systemReserved:
  cpu: "250m"
  memory: "256Mi"
evictionHard:
  memory.available: "200Mi"
  nodefs.available: "10%"

Bootstrap First Control-Plane Node

# Pre-flight: disable swap, load kernel modules, set sysctl
swapoff -a
modprobe overlay br_netfilter
sysctl -w net.bridge.bridge-nf-call-iptables=1
sysctl -w net.ipv4.ip_forward=1

# Install containerd, kubeadm, kubelet, kubectl
# (distro-specific — see Kubernetes docs)

# Init first control-plane
kubeadm init --config kubeadm-config.yaml --upload-certs

# Output includes:
# kubeadm join k8s-api.example.com:6443 \
#   --token abcdef.1234567890abcdef \
#   --discovery-token-ca-cert-hash sha256:... \
#   --control-plane --certificate-key <cert-key>

Join Remaining Control-Plane Nodes

# On each additional control-plane node:
kubeadm join k8s-api.example.com:6443 \
  --token abcdef.1234567890abcdef \
  --discovery-token-ca-cert-hash sha256:abc123... \
  --control-plane \
  --certificate-key <cert-key> \
  --apiserver-advertise-address 10.0.0.2   # this node's IP

Join Worker Nodes

# On each worker node:
kubeadm join k8s-api.example.com:6443 \
  --token abcdef.1234567890abcdef \
  --discovery-token-ca-cert-hash sha256:abc123...

# Verify cluster
kubectl get nodes -o wide
# NAME        STATUS   ROLES           AGE   VERSION   INTERNAL-IP
# cp-1        Ready    control-plane   10m   v1.29.0   10.0.0.1
# cp-2        Ready    control-plane   8m    v1.29.0   10.0.0.2
# cp-3        Ready    control-plane   7m    v1.29.0   10.0.0.3
# worker-1    Ready    <none>          5m    v1.29.0   10.0.1.10
# worker-2    Ready    <none>          5m    v1.29.0   10.0.1.11

Post-Install Steps

Install CNI plugin (kubectl apply -f cilium.yaml)
Install metrics-server
Configure default StorageClass
Install cert-manager + cluster issuers
Install ingress controller
Configure OIDC authentication for human users
Apply default LimitRange and ResourceQuota to namespaces
Enable audit logging to a SIEM
Set up Prometheus + Grafana for cluster monitoring
Schedule etcd backups (hourly to object storage)

✅ Production Readiness Checklist

Before declaring a cluster production-ready, verify every item below:

Control Plane & etcd

3+ control-plane nodes across 3 AZs
Load balancer in front of API server (not a raw node IP)
etcd on local NVMe/SSD — verified fsync latency < 10ms
etcd backups automated hourly, stored cross-region, restore tested
Certificate rotation enabled (rotateCertificates: true)
API server audit logging enabled and shipped to SIEM

Networking

CNI installed and all nodes Ready
NetworkPolicy default-deny applied to all namespaces
kube-proxy in IPVS or eBPF mode (not iptables for >100 Services)
Pod CIDR does not overlap with node CIDR or on-prem networks
DNS (CoreDNS) scaled to at least 2 replicas with PDB

Security

RBAC enabled (default since 1.8 — verify --authorization-mode includes RBAC)
PodSecurity admission enabled with at minimum baseline in all namespaces
No pods running as root unless explicitly required
Secrets encrypted at rest (--encryption-provider-config)
OIDC/SSO configured for human users — no sharing of admin kubeconfig
Image vulnerability scanning enforced (Harbor / Trivy)

Observability

Prometheus + Grafana installed with kube-state-metrics and node-exporter
Alerts configured for: node NotReady, PVC pending, high CPU/memory, etcd latency
Centralised logging (Loki / Elasticsearch / CloudWatch)
Distributed tracing configured for critical services

Workload Defaults

Default LimitRange in every namespace (prevents unbounded resource usage)
ResourceQuota on all team namespaces
PodDisruptionBudget on all critical Deployments and StatefulSets
All Deployments have liveness + readiness probes
All containers have resource requests and limits set

📝 Knowledge Check

Q1. You are building a production cluster. What is the minimum number of control-plane nodes required to tolerate one control-plane node failure while maintaining etcd quorum?

A) 1 node — single-node control plane is enough for production
B) 2 nodes — one active, one standby
C) 3 nodes — quorum requires (3/2)+1 = 2 nodes to agree
D) 5 nodes — etcd requires 5 for production

✅ C) 3 nodes. A 3-node etcd cluster requires 2 members for quorum (majority). Losing 1 node still leaves 2 — quorum is maintained and writes continue. 2-node clusters have no fault tolerance (losing either node breaks quorum). 5 nodes is used when you need to tolerate 2 simultaneous failures.

Q2. All your kubectl commands suddenly return "connection refused" but the nodes and pods are healthy. What is the most likely root cause?

A) All kubelets have crashed simultaneously
B) The load balancer in front of the API server is down or misconfigured
C) etcd is full — the cluster is in read-only mode
D) The CNI plugin has crashed, blocking API traffic

✅ B) The load balancer is down. In an HA setup all clients connect to the LB VIP/DNS. If the LB fails, kubectl and all controllers lose API access even though the API server pods are healthy. This is why the LB must itself be HA (cloud NLB, keepalived VIP, etc.) — it is the single most critical component in the data path.

Q3. A developer's pod uses 10 GB of memory but their namespace has no LimitRange or ResourceQuota. What risk does this create?

A) No risk — Kubernetes automatically limits pod memory to the node capacity
B) The pod can consume all memory on its node, triggering OOM eviction of other pods
C) The scheduler will reject the pod if no LimitRange is set
D) The pod will be QoS class Guaranteed and protected from eviction

✅ B) OOM eviction of other pods. Without a limit, the pod is QoS class BestEffort — it has no guaranteed allocation but also no ceiling. It can consume all available node memory, causing the kernel OOM killer to kill other pods on the same node. Always set a default LimitRange to prevent this in every namespace.