Scheduler Framework — Kubernetes Mastery

🔄 The Scheduling Pipeline

The Kubernetes scheduler is a single-threaded control loop that watches for unscheduled Pods (spec.nodeName == "") and assigns them to nodes. Since Kubernetes 1.19 it is built entirely on the Scheduling Framework — a plugin API with well-defined extension points at every stage of the cycle.

The cycle has two phases: Scheduling (pick a node) and Binding (commit the assignment).

Filter (hard constraints)

Eliminates nodes that cannot run the Pod: insufficient resources, taints, node selectors, affinity rules, volume topology.

Score (soft preferences)

Ranks remaining nodes 0–100. The node with the highest weighted sum wins. Plugins: least-requested, image locality, topology spread.

Reserve + Permit

Reserve claims resources optimistically. Permit can delay binding (e.g. gang scheduling — wait for all pods of a group to be schedulable).

Bind

Writes spec.nodeName to the Pod via the API server. The Kubelet on the target node then picks it up and starts the containers.

🔌 Extension Points Reference

Extension Point	Phase	Purpose	Return
`QueueSort`	pre-sched	Determines priority order of the scheduling queue	less/greater
`PreFilter`	sched	Pre-process Pod info, cache data for Filter plugins	error or nil
`Filter`	sched	Eliminate infeasible nodes (hard constraints)	pass or reject
`PostFilter`	sched	Called when Filter eliminates all nodes — implements preemption	nominated node
`PreScore`	sched	Pre-process for Score plugins, shared state	error or nil
`Score`	sched	Assign 0–100 score to each feasible node	int64 score
`NormalizeScore`	sched	Normalise plugin scores to 0–100 range	adjusted scores
`Reserve`	sched	Optimistically claim resources before binding	error or nil
`Permit`	sched	Allow, deny, or wait (gang scheduling)	allow/deny/wait
`PreBind`	bind	Prepare resources before bind (e.g. provision PV)	error or nil
`Bind`	bind	Write `spec.nodeName` to the API server	error or nil
`PostBind`	post	Informational — cleanup after successful bind	void

Built-in Filter Plugins

Plugin	What it filters
`NodeUnschedulable`	Nodes with `spec.unschedulable: true` (`kubectl cordon`)
`NodeResourcesFit`	Nodes with insufficient CPU/memory/extended resources
`NodeName`	Nodes that don't match `spec.nodeName` (if set)
`NodeAffinity`	Nodes that don't match `nodeAffinity` required rules
`TaintToleration`	Nodes with taints the Pod doesn't tolerate
`VolumeBinding`	Nodes where required PVCs can't be bound (topology constraints)
`InterPodAffinity`	Nodes violating required inter-pod affinity/anti-affinity
`NodePorts`	Nodes where required host ports are already in use

Built-in Score Plugins

Plugin	What it prefers	Weight
`LeastAllocated`	Nodes with most free CPU+memory (spread workloads)	1
`MostAllocated`	Nodes with least free resources (bin-pack)	disabled by default
`NodeAffinity`	Nodes matching preferred affinity rules	2
`InterPodAffinity`	Nodes co-located with preferred pods	2
`TaintToleration`	Nodes with fewer tolerated taints (prefer clean nodes)	1
`ImageLocality`	Nodes that already have the container image cached	1
`TopologySpreadConstraint`	Nodes that minimise topology skew	2

ℹ️ Weighted score formula Final node score = Σ (plugin_score × plugin_weight) / Σ weights. Plugins can have different weights via KubeSchedulerConfiguration. The node with the highest final score wins; ties are broken randomly.

⚡ Preemption & KubeSchedulerConfiguration

Preemption (PostFilter)

When the Filter phase eliminates all nodes for a high-priority Pod, the PostFilter extension point runs. The built-in DefaultPreemption plugin finds nodes where evicting lower-priority Pods would make room, then marks those Pods for eviction (nominatedNodeName) and retries scheduling.

# High-priority Pod — will preempt lower-priority Pods if no node fits
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
preemptionPolicy: PreemptLowerPriority   # default
globalDefault: false
---
apiVersion: v1
kind: Pod
metadata:
  name: critical-job
spec:
  priorityClassName: high-priority
  containers:
    - name: app
      image: myapp:latest
      resources:
        requests: { cpu: "4", memory: "8Gi" }

⚠️ Preemption is disruptive Preempted Pods receive a graceful termination signal — their work is interrupted. Use preemptionPolicy: Never on PriorityClasses where eviction is not acceptable (e.g. long-running batch jobs).

KubeSchedulerConfiguration

Configure plugin weights, enable/disable plugins, and define multiple scheduler profiles via KubeSchedulerConfiguration:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
    plugins:
      score:
        disabled:
          - name: LeastAllocated      # disable spread
        enabled:
          - name: MostAllocated       # enable bin-packing
            weight: 3
      filter:
        disabled:
          - name: NodeResourcesFit    # replace with custom version
        enabled:
          - name: CustomResourcesFit
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated       # or LeastAllocated, RequestedToCapacityRatio
            resources:
              - name: cpu
                weight: 1
              - name: memory
                weight: 1
              - name: "example.com/gpu"
                weight: 2             # weight GPU more heavily in scoring

  # Second profile — used by GPU-intensive workloads
  - schedulerName: gpu-scheduler
    plugins:
      filter:
        enabled:
          - name: GPUFilter
      score:
        enabled:
          - name: GPUScore
            weight: 5

💡 Multiple scheduler profiles A single kube-scheduler binary can run multiple scheduler profiles. Pods select a profile by setting spec.schedulerName: gpu-scheduler. No separate scheduler deployment needed.

Permit — Gang Scheduling

The Permit extension point lets a plugin return Wait instead of proceeding immediately. This enables gang scheduling — holding all Pods of a job until enough nodes are simultaneously available.

# Coscheduling plugin (from scheduler-plugins repo)
# Pods in the same PodGroup must all be schedulable simultaneously
apiVersion: scheduling.sigs.k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: ml-training-job
spec:
  minMember: 8        # all 8 pods must be schedulable before any is bound
  minResources:
    cpu: "32"
    memory: "128Gi"

🛠️ Writing a Custom Scheduler Plugin

Custom plugins are compiled into a scheduler binary (or run as a second profile). They implement one or more framework interfaces in Go.

// Example: a Filter plugin that rejects nodes missing a custom label
package myplugin

import (
    "context"
    "fmt"
    v1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "sigs.k8s.io/scheduler-plugins/apis/config"
    "k8s.io/kubernetes/pkg/scheduler/framework"
)

const Name = "RequireLabel"

type RequireLabel struct{}

// Implement the framework.FilterPlugin interface
func (r *RequireLabel) Name() string { return Name }

func (r *RequireLabel) Filter(
    ctx context.Context,
    state *framework.CycleState,
    pod *v1.Pod,
    nodeInfo *framework.NodeInfo,
) *framework.Status {
    requiredLabel, ok := pod.Annotations["require-node-label"]
    if !ok {
        return framework.NewStatus(framework.Success)  // no constraint
    }
    node := nodeInfo.Node()
    if _, exists := node.Labels[requiredLabel]; !exists {
        return framework.NewStatus(
            framework.Unschedulable,
            fmt.Sprintf("node %s missing required label %s", node.Name, requiredLabel),
        )
    }
    return framework.NewStatus(framework.Success)
}

// Register the plugin in main.go
func New(_ runtime.Object, h framework.Handle) (framework.Plugin, error) {
    return &RequireLabel{}, nil
}

// main.go — embed custom plugin into scheduler
package main

import (
    "sigs.k8s.io/scheduler-plugins/pkg/myplugin"
    "k8s.io/kubernetes/cmd/kube-scheduler/app"
)

func main() {
    command := app.NewSchedulerCommand(
        app.WithPlugin(myplugin.Name, myplugin.New),
    )
    command.Execute()
}

Debugging Scheduling Decisions

# Why is a Pod Pending? Get scheduler events
kubectl describe pod <pending-pod> | grep -A 10 Events

# Get detailed scheduling failure reasons (verbose logs)
kubectl logs -n kube-system deployment/kube-scheduler \
  | grep -i "failed\|filtered\|preempt"

# Use scheduler extender debug endpoint (if enabled)
# GET /scheduler/v1/extenders/status

# Simulate scheduling without committing (dry-run node scoring)
# Temporarily set --v=10 on kube-scheduler for full trace

Scheduler Plugins Repo

sigs.k8s.io/scheduler-plugins — community plugins: coscheduling, capacity scheduling, node resources topology, trimaran (load-aware).

Load-Aware Scheduling

Trimaran plugin uses real-time CPU/memory utilisation (from metrics-server) to score nodes — avoids hot spots that pure request-based scoring misses.

Topology-Aware Scheduling

Node Resource Topology plugin reads NUMA topology and CPU/device affinity to schedule latency-sensitive workloads on optimal NUMA nodes.

Second Scheduler Profile

Run a bin-packing profile (MostAllocated) for batch jobs and a spreading profile (LeastAllocated) for services — same binary, two profiles.

📝 Knowledge Check

Q1. A Pod is Pending with the message "0/5 nodes are available: 5 Insufficient memory." Which extension point eliminated all nodes?

A) Score — NodeAffinity gave all nodes a score of 0
B) Filter — NodeResourcesFit rejected all nodes
C) Permit — gang scheduling is waiting for sibling pods
D) PostFilter — preemption failed to find a victim

✅ B) Filter — NodeResourcesFit. "Insufficient memory" means the NodeResourcesFit filter plugin rejected every node because none had enough free memory to satisfy the Pod's resources.requests.memory. Increase the request, reduce it, or add more nodes.

Q2. You want kube-scheduler to bin-pack Pods onto as few nodes as possible to save cost. Which built-in Score plugin should you enable?

A) LeastAllocated — prefers nodes with the most free resources
B) ImageLocality — prefers nodes with cached images
C) MostAllocated — prefers nodes with the least free resources
D) TopologySpreadConstraint — spreads Pods across zones

✅ C) MostAllocated. MostAllocated scores nodes higher when they are more heavily utilised — this packs Pods onto fewer nodes, allowing underutilised nodes to be drained and removed (cluster autoscaler scale-down). It is disabled by default; swap it in via KubeSchedulerConfiguration.

Q3. What is the purpose of the Permit extension point, and which scheduling pattern does it enable?

A) It permanently rejects a Pod if no node is found — same as Filter
B) It allows a plugin to delay binding until a condition is met — enables gang scheduling
C) It authorises the scheduler to write to the API server
D) It cleans up reserved resources if binding fails

✅ B) Delay binding — gang scheduling. The Permit extension point returns Wait to hold a Pod in a waiting state after a node is selected but before Bind. The coscheduling plugin uses this to hold all Pods of a job until the minimum member count is simultaneously schedulable, preventing partial job startup.