kube-proxy Modes — Kubernetes Mastery

🌐 What kube-proxy Does

A Kubernetes Service gets a stable virtual IP (ClusterIP) that does not correspond to any network interface. kube-proxy watches Services and EndpointSlices and programs the Linux kernel's packet-forwarding subsystem so that traffic to the ClusterIP is load-balanced to healthy pod IPs.

kube-proxy itself does not sit in the data path — it only programs rules. Actual packet forwarding happens in the kernel (iptables/IPVS) or via eBPF programs loaded by the CNI plugin (Cilium).

🟡 iptables

Default mode since K8s 1.2
Random probabilistic LB
O(n) rule scan per packet
Scales to ~1000 Services

🔵 IPVS

Default in many managed K8s
Hash table lookup O(1)
Multiple LB algorithms
Scales to 10,000+ Services

🟢 eBPF (Cilium)

Replaces kube-proxy entirely
Socket-level bypass (no NAT)
Lowest latency, highest scale
Requires Cilium CNI

🟡 iptables Mode

In iptables mode kube-proxy programs PREROUTING and OUTPUT chains in the nat table. For each Service it creates a chain of rules that randomly DNATs traffic to one of the backend pod IPs with probability-weighted --probability matches.

Rule Structure

# kube-proxy creates chains like:
# KUBE-SERVICES  → jumps to per-service chains
# KUBE-SVC-XXXX  → per-service: selects a backend
# KUBE-SEP-YYYY  → per-endpoint: DNATs to pod IP:port

# Inspect the rules (on a node)
iptables -t nat -L KUBE-SERVICES -n | head -20
iptables -t nat -L KUBE-SVC-I4BDJWRJOH4JFNXD -n
# Chain KUBE-SVC-I4BDJWRJOH4JFNXD (1 references)
# target       prot opt source    destination
# KUBE-SEP-AA  all  --  0.0.0.0/0 0.0.0.0/0  /* my-svc */ statistic mode random probability 0.33333
# KUBE-SEP-BB  all  --  0.0.0.0/0 0.0.0.0/0  /* my-svc */ statistic mode random probability 0.50000
# KUBE-SEP-CC  all  --  0.0.0.0/0 0.0.0.0/0  /* my-svc */

# Each SEP chain does the actual DNAT
iptables -t nat -L KUBE-SEP-AABBCCDD -n
# DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp to:10.0.2.11:80

ℹ️ Probability chaining for equal distribution For N endpoints: first rule has probability 1/N, second 1/(N-1), third 1/(N-2)… This gives equal distribution because each rule is only reached if earlier ones didn't match.

Conntrack & Session Affinity

iptables mode relies on Linux conntrack to ensure packets of the same TCP connection all go to the same pod (the DNAT decision is remembered per-flow). Session affinity (sessionAffinity: ClientIP) adds an extra recent match module to stick a client IP to one backend for the configured timeout.

# Enable session affinity on a Service
apiVersion: v1
kind: Service
spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800   # 3 hours (default)

iptables Scalability Problem

Every packet must traverse the KUBE-SERVICES chain linearly until it matches. With 10,000 Services (100,000 rules), this causes:

High per-packet CPU cost (O(n) linear scan)
Slow rule programming — kube-proxy must rewrite all rules atomically on any endpoint change
conntrack table exhaustion under heavy connection rates

# Check iptables rule count
iptables -t nat -L | wc -l

# Check conntrack table usage
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# Conntrack exhaustion causes "nf_conntrack: table full, dropping packet"
# Increase limit:
sysctl -w net.netfilter.nf_conntrack_max=524288

⚠️ iptables mode breaks down beyond ~1000 Services If your cluster has more than ~1000 Services or endpoints, switch to IPVS or eBPF mode. The O(n) rule evaluation causes measurable latency spikes and kube-proxy CPU saturation during rapid endpoint churn.

🔵 IPVS Mode

IPVS (IP Virtual Server) is a Linux kernel load balancer built into the netfilter framework. kube-proxy in IPVS mode creates a virtual server per Service ClusterIP and adds real servers (pod IPs) as backends. Lookups use a hash table — O(1) regardless of cluster size.

Load-Balancing Algorithms

Algorithm	Flag	Best for
Round Robin	`rr` (default)	Homogeneous backends, equal request cost
Least Connection	`lc`	Variable request duration — avoids hot backends
Source Hash	`sh`	Session affinity by source IP (stateless implementation)
Destination Hash	`dh`	Cache-friendly — same destination always goes to same backend
Shortest Expected Delay	`sed`	Weighted least connection with request overhead

Enable IPVS Mode

# Ensure kernel modules are loaded on each node
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack

# kube-proxy ConfigMap (edit in-place)
kubectl edit configmap kube-proxy -n kube-system
# Under data.config.conf:
#   mode: "ipvs"
#   ipvs:
#     scheduler: "lc"     # least connection

# Restart kube-proxy DaemonSet
kubectl rollout restart daemonset kube-proxy -n kube-system

# Verify on a node
ipvsadm -L -n | head -30
# IP Virtual Server version 1.2.1
# Prot LocalAddress:Port Scheduler Flags
#   -> RemoteAddress:Port  Forward Weight ActiveConn InActConn
# TCP  10.96.0.10:80 lc
#   -> 10.0.2.11:80        Masq    1      0          0
#   -> 10.0.3.22:80        Masq    1      0          0

💡 IPVS still uses iptables for SNAT IPVS handles DNAT (ClusterIP → pod) but still uses iptables/netfilter for SNAT (masquerade) on the return path. It is not a full iptables replacement — it just does the load-balancing lookup in O(1).

🟢 eBPF Mode — Cilium kube-proxy Replacement

Cilium can completely replace kube-proxy. Instead of programming iptables/IPVS rules, Cilium attaches eBPF programs at the socket layer — packets are redirected to the correct pod before they even enter the network stack, eliminating NAT entirely for same-node traffic.

Why eBPF Is Faster

Socket-level DNAT

For same-node pod→service traffic, eBPF intercepts at the socket connect() call — the packet is redirected before leaving the process. Zero NAT overhead.

No conntrack for ClusterIP

eBPF bypasses netfilter conntrack for Service traffic. Reduces CPU and memory pressure — critical for high connection-rate workloads.

O(1) BPF Map Lookups

Service backends are stored in BPF hash maps. Lookup is O(1) regardless of Service count — handles 100k+ Services with no degradation.

Preserves Source IP

eBPF can preserve the original client IP all the way to the pod without SNAT — simplifying access logs and IP-based policies.

Enable Cilium kube-proxy Replacement

# Install Cilium without kube-proxy (set kubeProxyReplacement=true)
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=<API_SERVER_IP> \
  --set k8sServicePort=6443

# Verify replacement is active
kubectl exec -n kube-system ds/cilium -- cilium status | grep KubeProxyReplacement
# KubeProxyReplacement: True

# View eBPF service map
kubectl exec -n kube-system ds/cilium -- cilium service list
# ID   Frontend             Service Type   Backend
# 1    10.96.0.10:80/TCP    ClusterIP      1 => 10.0.2.11:80
#                                          2 => 10.0.3.22:80

⚠️ Requires Linux kernel ≥ 5.10 + BTF Cilium eBPF mode requires a modern kernel with BTF (BPF Type Format) support. Ensure nodes run kernel 5.10+ (Ubuntu 22.04 LTS, Debian 11+, RHEL 9, or Amazon Linux 2023 are all compatible).

⚖️ Mode Comparison & When to Use Each

Feature	iptables	IPVS	eBPF (Cilium)
Lookup complexity	O(n) per packet	O(1) hash table	O(1) BPF map
Rule update speed	Full rewrite on change	Incremental	Incremental BPF map
Max Services (practical)	~1,000	~10,000+	100,000+
LB algorithms	Random only	rr, lc, sh, dh, sed…	Configurable (maglev etc.)
NAT for ClusterIP	iptables DNAT	IPVS DNAT + iptables SNAT	Socket-level (no NAT same-node)
Conntrack required	Yes	Yes (SNAT)	No (for ClusterIP)
Source IP preservation	No (SNAT)	No (SNAT)	Yes (option)
Kernel requirement	Any	Any + ip_vs modules	≥ 5.10 + BTF
Replaces kube-proxy	No	No	Yes
Best for	Small clusters (<50 nodes)	Medium–large clusters	Large/enterprise, low latency

NodePort & LoadBalancer Internals

For NodePort Services, kube-proxy additionally programs a rule in the KUBE-NODEPORTS chain to DNAT traffic arriving on the node's port nodePort (30000–32767) to the service backends — on every node, regardless of where the pods run.

# NodePort rule — catches traffic on any node on port 32080
iptables -t nat -L KUBE-NODEPORTS -n
# KUBE-SVC-XXXX tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:32080

# ExternalTrafficPolicy: Local — only route to pods on THIS node
# Avoids a second hop but causes unequal load if pods are unevenly distributed
apiVersion: v1
kind: Service
spec:
  type: NodePort
  externalTrafficPolicy: Local   # preserves client source IP, no SNAT

ℹ️ externalTrafficPolicy: Local With Local, kube-proxy only adds backend rules for pods on the same node. Health checks from the load balancer will fail on nodes with no local pods — the cloud LB stops routing to those nodes. This preserves source IPs and eliminates cross-node hops at the cost of potentially uneven load.

Debugging Service Connectivity

# Test ClusterIP reachability from inside a pod
kubectl run debug --image=nicolaka/netshoot --rm -it -- /bin/bash
curl http://<service-name>.<namespace>.svc.cluster.local

# Check EndpointSlices (is the service backed by any pods?)
kubectl get endpointslices -l kubernetes.io/service-name=my-svc
kubectl describe endpointslices -l kubernetes.io/service-name=my-svc

# Check iptables rules on a node
ssh node-1 "iptables -t nat -S | grep KUBE | wc -l"

# Check IPVS virtual servers
ssh node-1 "ipvsadm -L -n | grep -A 5 <ClusterIP>"

# Check kube-proxy logs
kubectl logs -n kube-system daemonset/kube-proxy | tail -50

📝 Knowledge Check

Q1. A cluster has 5,000 Services. kube-proxy is running in iptables mode and ops reports high CPU on all nodes. What is the most likely cause and fix?

A) Too many pods — reduce replica count to lower CPU usage
B) iptables O(n) rule scanning — switch kube-proxy to IPVS or eBPF mode
C) The API server is sending too many watch events
D) conntrack table is too small — increase nf_conntrack_max

✅ B) iptables O(n) scanning — switch to IPVS. At 5,000 Services, iptables generates tens of thousands of rules. Every packet must scan the chain linearly until it matches, causing high CPU. IPVS uses a hash table lookup (O(1)) and is the standard fix for large clusters.

Q2. A Service is configured with externalTrafficPolicy: Local. A client connects to node-3 but all pods for this service run on node-1 and node-2. What happens?

A) Traffic is forwarded from node-3 to node-1 transparently
B) Traffic is dropped — no local backend exists on node-3
C) kube-proxy temporarily falls back to Cluster policy on node-3
D) The connection hangs until a pod is scheduled on node-3

✅ B) Traffic is dropped. With externalTrafficPolicy: Local, kube-proxy only programs backends for pods on the same node. Node-3 has no backend rules for this service — connections are dropped. The cloud load balancer health check will fail on node-3, causing it to stop routing traffic there (by design).

Q3. What is the key performance advantage of Cilium eBPF mode over IPVS for same-node pod→Service traffic?

A) eBPF uses faster network cards
B) eBPF intercepts at socket connect() — no packet NAT, no conntrack for ClusterIP traffic
C) eBPF has a larger conntrack table than IPVS
D) eBPF eliminates the need for DNS resolution

✅ B) Socket-level interception — no NAT, no conntrack. Cilium eBPF intercepts at the socket layer when the application calls connect(). The kernel rewrites the destination before any packet is formed — so the packet goes directly to the pod IP with zero NAT processing and no conntrack entry needed. This is the fundamental latency and CPU advantage over IPVS.