The Container Network Interface (CNI) is the plugin specification that defines how networking is set up for Pods. The kubelet doesn't know how to configure networking — it delegates entirely to a CNI plugin. Understanding CNI is essential for troubleshooting node-level networking issues and making architectural decisions about cluster networking.

1. The CNI Specification

CNI is deliberately minimal — a spec for executing a binary that sets up (or tears down) networking for a container. The kubelet calls CNI at two points in a Pod's life:

How It's Invoked

File Locations on a Node

# CNI plugin binaries:
ls /opt/cni/bin/
# bridge  calico  flannel  host-local  loopback  portmap  ...

# CNI configuration (kubelet reads first file alphabetically):
ls /etc/cni/net.d/
# 10-calico.conflist    (or 10-flannel.conflist, etc.)

CNI Config Example

# /etc/cni/net.d/10-calico.conflist
{
  "name": "k8s-pod-network",
  "cniVersion": "1.0.0",
  "plugins": [
    {
      "type": "calico",              // Plugin binary name
      "ipam": {
        "type": "calico-ipam"        // IP Address Management plugin
      },
      "policy": {
        "type": "k8s"
      }
    },
    {
      "type": "portmap",             // Chained plugin: port mapping for hostPort
      "capabilities": {"portMappings": true}
    },
    {
      "type": "bandwidth",           // Chained plugin: traffic shaping
      "capabilities": {"bandwidth": true}
    }
  ]
}

IPAM — IP Address Management

2. CNI Plugins: Deep Comparison

Flannel — Simple Overlay

Calico — Flexible & Enterprise-Ready

# Calico in BGP mode — no overlay, native routing:
# Node 1 announces: "10.244.1.0/24 via 192.168.1.10"
# Node 2 announces: "10.244.2.0/24 via 192.168.1.11"
# Traffic flows directly via learned routes — no encapsulation

Cilium — eBPF-Powered

Head-to-Head Comparison

3. eBPF — The Modern Data Plane

eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs inside the Linux kernel without modifying kernel source or loading kernel modules. For networking, this means:

Traditional (iptables) vs eBPF

What eBPF Enables in Cilium

# Cilium without kube-proxy:
# Install Cilium with kube-proxy replacement:
helm install cilium cilium/cilium \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=API_SERVER_IP \
  --set k8sServicePort=6443

# Verify kube-proxy is not needed:
kubectl -n kube-system delete ds kube-proxy
# Cilium handles all Service routing via eBPF

4. CNI Troubleshooting

Common Issues

Key Debug Commands

# Check CNI DaemonSet health:
kubectl get pods -n kube-system -l k8s-app=calico-node   # Calico
kubectl get pods -n kube-system -l app.kubernetes.io/name=cilium  # Cilium

# CNI plugin logs:
kubectl logs -n kube-system -l k8s-app=calico-node --tail=50
kubectl logs -n kube-system -l app.kubernetes.io/name=cilium --tail=50

# Cilium-specific:
kubectl exec -n kube-system cilium-xxxxx -- cilium status
kubectl exec -n kube-system cilium-xxxxx -- cilium endpoint list

# Check node networking:
kubectl debug node/worker-1 -it --image=nicolaka/netshoot -- bash
  # Inside: ip route, ip link, bridge fdb, iptables-save

# Calico-specific:
kubectl exec -n kube-system calico-node-xxxxx -- calicoctl node status
kubectl exec -n kube-system calico-node-xxxxx -- calicoctl get ippool

5. Choosing a CNI for Your Cluster

# Decision tree:
Need NetworkPolicy?
├─ NO → Flannel (simplest, lowest resource)
└─ YES → Need L7 policies or replace kube-proxy?
          ├─ NO → Calico (battle-tested, BGP or VXLAN)
          └─ YES → Cilium (eBPF, L7 visibility, kube-proxy replacement)

Running on AWS EKS?
└─ Default: AWS VPC CNI (native VPC IPs, best integration)
   └─ Add Calico for NetworkPolicy enforcement

Running on GKE?
└─ Default: GKE Dataplane v2 (Cilium-based, eBPF)

Bare metal / on-prem?
└─ Calico with BGP (native routing, no overlay overhead)
   or Cilium (eBPF + Hubble observability)

Summary