1. Recap: Single-Host Networking

On a single Docker host, container networking is straightforward:

• veth pairs — virtual Ethernet cables connecting each container to a bridge
• docker0 bridge — a software switch that forwards frames between containers
• iptables NAT — masquerades outbound traffic and routes published ports

This works perfectly on one machine. But real production systems span dozens or hundreds of hosts. How do containers on different hosts communicate as if they're on the same LAN?

2. Overlay Networks

The problem: Container A on Host 1 (IP 10.0.1.5) needs to reach Container B on Host 2 (IP 10.0.1.9). The underlying physical network only knows about host IPs — it has no route to 10.0.1.x.

The solution: Encapsulate the container-to-container packet inside a host-to-host packet. The physical network delivers the outer packet; the destination host unwraps it and delivers the inner packet to the target container.

VXLAN (Virtual eXtensible LAN) is the dominant overlay protocol. It wraps full Layer 2 Ethernet frames inside UDP packets (port 4789).

3. How VXLAN Works

• VTEP (Virtual Tunnel Endpoint) — a kernel-level component on each host that handles encapsulation and decapsulation.
• ARP resolution — when Container A needs Container B's MAC, the VTEP resolves it across hosts (via multicast, unicast flooding, or a control plane lookup).
• Transparent to containers — containers see a flat L2 network; they don't know packets are being wrapped.

Performance Costs

4. CNI Plugins (Kubernetes)

CNI (Container Network Interface) is the Kubernetes standard for pluggable networking. Each plugin implements pod-to-pod connectivity differently:

5. The eBPF Revolution

eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs inside the Linux kernel without modifying kernel source or loading kernel modules.

For Networking, eBPF Replaces iptables

• iptables problem: rules are evaluated sequentially — O(n). At thousands of services, rule traversal becomes a bottleneck.
• eBPF solution: hash-map lookups — O(1). Programmable packet processing with zero-copy.

What Cilium Uses eBPF For

• Pod-to-pod routing
• Service load balancing (replaces kube-proxy)
• Network policy enforcement
• Observability (Hubble)

6. Network Policies

Network policies are firewall rules for containers — specifying which pods can talk to which.

# Kubernetes NetworkPolicy: only allow "api" pods to reach "db" on port 5432
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-allow-api-only
spec:
  podSelector:
    matchLabels:
      app: db
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 5432

Key points:

• Default: all traffic allowed (no policies = open network)
• Once a policy selects a pod, only explicitly allowed traffic is permitted (default-deny for that pod)
• Calico and Cilium enforce policies at the kernel level (eBPF or iptables) — not in userspace proxies

7. Service Mesh Networking

A service mesh adds a sidecar proxy (typically Envoy) next to every pod, intercepting all inbound and outbound traffic.

Capabilities

• mTLS — mutual TLS between every service (zero-trust networking)
• Observability — every request is traced, metered, logged
• Traffic splitting — canary deployments by percentage (e.g., 5% to v2)
• Retries & circuit breaking — resilience without code changes

Architecture

Hands-On: Docker Overlay Network Across Swarm Nodes

# On manager node: initialize swarm
docker swarm init --advertise-addr <MANAGER_IP>

# On worker node: join swarm (use token from init output)
docker swarm join --token <TOKEN> <MANAGER_IP>:2377

# Create overlay network
docker network create --driver overlay --attachable my-overlay

# On manager: run a container on the overlay
docker run -dit --name web --network my-overlay alpine sleep 3600

# On worker: run another container on the same overlay
docker run -dit --name client --network my-overlay alpine sleep 3600

# From client, ping web by container name (DNS resolves across hosts!)
docker exec client ping -c 3 web

# Inspect the network — see both containers
docker network inspect my-overlay

# Observe VXLAN in action (on either host):
sudo tcpdump -i eth0 udp port 4789 -c 5

Industry Callout

Knowledge Check

Key Takeaways

• Overlay networks (VXLAN) solve multi-host container connectivity by encapsulating traffic in host-to-host packets.
• CNI plugins offer different trade-offs: simplicity (Flannel), performance (Calico/BGP), programmability (Cilium/eBPF).
• eBPF is revolutionizing Linux networking — O(1) lookups replace O(n) iptables chains, enabling scalable routing, load balancing, and policy.
• Network policies provide container-level firewalling — essential for zero-trust architectures.
• Service meshes add mTLS, observability, and traffic control via sidecar proxies — powerful but add complexity.
• Modern production clusters combine these layers: Cilium for CNI + eBPF networking, with optional service mesh for L7 features.