Taints and tolerations work together to repel Pods from nodes. A taint on a node says "don't schedule here unless you explicitly tolerate me." A toleration on a Pod says "I can handle that taint." This is the opposite of node affinity — affinity attracts Pods to nodes; taints repel Pods from nodes.

1. How Taints & Tolerations Work

Taint Format

# Taint = key=value:effect
# Three parts: key, value (optional), effect

# Apply a taint:
kubectl taint nodes worker-3 gpu=true:NoSchedule

# Remove a taint (trailing minus):
kubectl taint nodes worker-3 gpu=true:NoSchedule-

# View taints:
kubectl describe node worker-3 | grep Taints
# Taints: gpu=true:NoSchedule

The Three Effects

2. Writing Tolerations

Exact Match

# Tolerates the specific taint gpu=true:NoSchedule
spec:
  tolerations:
    - key: "gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

Exists Operator (Key-Only Match)

# Tolerates ANY taint with key "gpu" regardless of value:
spec:
  tolerations:
    - key: "gpu"
      operator: "Exists"
      effect: "NoSchedule"

# Tolerates ALL taints with key "gpu" (any effect):
    - key: "gpu"
      operator: "Exists"         # no effect specified = matches all effects

# Tolerates ALL taints on ALL keys (nuclear option):
    - operator: "Exists"         # no key = matches everything

Toleration with tolerationSeconds (for NoExecute)

# "I can tolerate this taint, but only for 300 seconds, then evict me"
spec:
  tolerations:
    - key: "node.kubernetes.io/unreachable"
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 300     # Stay for 5 min, then get evicted

Matching Rules

# Summary of matching:
# Taint: dedicated=gpu:NoSchedule
#
# Toleration matches if:
# key=dedicated, operator=Equal, value=gpu, effect=NoSchedule  ✓
# key=dedicated, operator=Exists, effect=NoSchedule            ✓
# key=dedicated, operator=Exists                               ✓ (any effect)
# operator=Exists                                              ✓ (matches all)
# key=dedicated, operator=Equal, value=cpu, effect=NoSchedule  ✗ (wrong value)
# key=other, operator=Equal, value=gpu, effect=NoSchedule      ✗ (wrong key)

3. Built-in Taints (Kubernetes-Managed)

Kubernetes automatically adds/removes these taints based on node conditions:

Default Tolerations on All Pods

Kubernetes automatically adds these tolerations to every Pod (unless overridden):

# All Pods tolerate not-ready and unreachable for 300s:
tolerations:
  - key: node.kubernetes.io/not-ready
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 300
  - key: node.kubernetes.io/unreachable
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 300

# This means: if a node goes NotReady, its Pods stay for 5 minutes
# before being evicted. Gives time for transient failures to recover.

4. Common Use Cases

Dedicated Nodes (GPU, High-Memory)

# Taint the GPU nodes:
kubectl taint nodes gpu-worker-1 dedicated=gpu:NoSchedule
kubectl taint nodes gpu-worker-2 dedicated=gpu:NoSchedule

# Only ML Pods tolerate it:
spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"
  nodeSelector:
    hardware: gpu                 # Also use nodeSelector to ATTRACT to GPU nodes
    
# Without nodeSelector, the Pod could land on ANY node
# Toleration only means "I'm allowed on GPU nodes" — not "I must go there"

Maintenance / Drain

# Cordon a node (adds unschedulable taint):
kubectl cordon worker-1
# Taint: node.kubernetes.io/unschedulable:NoSchedule

# Drain (cordon + evict all Pods):
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

# DaemonSets tolerate the unschedulable taint automatically
# (that's why --ignore-daemonsets is needed for drain)

Spot/Preemptible Nodes

# Cloud providers taint spot instances:
# Taint: cloud.google.com/gke-spot=true:NoSchedule (GKE)
# Taint: node.kubernetes.io/preemptible=true:NoSchedule

# Only cost-tolerant workloads should schedule there:
spec:
  tolerations:
    - key: "cloud.google.com/gke-spot"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

Summary