🔗 ownerReferences — The GC Graph

Kubernetes Garbage Collection (GC) works by following a directed graph of ownerReferences. Every object can declare one or more owners. When an owner is deleted, the GC controller deletes all objects that reference it — cascading through the graph.

Deployment ReplicaSet (old) ReplicaSet (cur) Pod A Pod B Pod C ownerRef → Deployment ownerRef → ReplicaSet (cur)

ownerReference fields

# The ownerReferences field on a ReplicaSet owned by a Deployment:
metadata:
  ownerReferences:
  - apiVersion:         apps/v1
    kind:               Deployment
    name:               my-app
    uid:                abc-123-def-456
    controller:         true   # only one owner may be controller=true
    blockOwnerDeletion: true   # foreground deletion waits for this
🔵 Rules for ownerReferences
  • Only one owner can have controller: true — prevents multiple controllers fighting over the same object.
  • Cross-namespace owner references are not allowed for namespaced resources. A Pod in namespace A cannot be owned by a resource in namespace B.
  • Cluster-scoped resources (e.g. PersistentVolume) cannot be owned by namespaced resources.
  • blockOwnerDeletion: true means foreground deletion waits for dependents to be deleted first.

Inspecting ownership

# See who owns a pod
kubectl get pod my-app-abc123 -o jsonpath='{.metadata.ownerReferences}'

# Find all resources owned by a Deployment (cascade preview)
kubectl get rs -l app=my-app -o custom-columns=\
  NAME:.metadata.name,\
  OWNER:.metadata.ownerReferences[0].name

# Check GC graph for a ReplicaSet
kubectl describe rs my-app-7d9f8 | grep -A3 "Controlled By"

🔒 Finalizer Patterns

Finalizers are strings in metadata.finalizers that act as deletion gates. An object with finalizers cannot be deleted from etcd until all finalizers are removed. Controllers are responsible for doing the cleanup work and then removing their finalizer.

Common finalizer use cases

External resource cleanup

Delete a cloud load balancer, S3 bucket, or DNS record before the Kubernetes object is removed.

Cross-namespace cleanup

Since ownerReferences can't cross namespaces, use a finalizer to clean up resources in other namespaces.

Backup before delete

Trigger a final backup of a database before the Database CR and its PVC are garbage collected.

Deregistration

Remove a service from a service registry (Consul, Eureka) or revoke credentials before the pod/resource disappears.

Well-known built-in finalizers

FinalizerWho sets itWhat it does
kubernetes.io/pvc-protectionPVC protection controllerPrevents PVC deletion while a Pod is actively using it
kubernetes.io/pv-protectionPV protection controllerPrevents PV deletion while bound to a PVC
foregroundDeletionAPI server (foreground cascade)Blocks owner deletion until all blocking dependents are gone
orphanAPI server (orphan cascade)Clears ownerReferences from dependents before owner is deleted
batch.kubernetes.io/job-trackingJob controllerTracks Pod completion before marking Job complete

Complete finalizer lifecycle in a controller

const myFinalizer = "cleanup.myorg.example.com"

func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    db := &myorgv1.Database{}
    if err := r.Get(ctx, req.NamespacedName, db); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // --- DELETION PATH ---
    if !db.DeletionTimestamp.IsZero() {
        if controllerutil.ContainsFinalizer(db, myFinalizer) {
            // Perform cleanup with a timeout
            cleanupCtx, cancel := context.WithTimeout(ctx, 60*time.Second)
            defer cancel()

            if err := r.deleteExternalResources(cleanupCtx, db); err != nil {
                // Update status to reflect cleanup failure
                db.Status.Phase = "TerminationFailed"
                r.Status().Update(ctx, db)
                return ctrl.Result{}, err
            }

            // Cleanup done — remove finalizer to allow deletion
            controllerutil.RemoveFinalizer(db, myFinalizer)
            if err := r.Update(ctx, db); err != nil {
                return ctrl.Result{}, err
            }
        }
        return ctrl.Result{}, nil  // object will be deleted
    }

    // --- CREATION PATH: add finalizer if missing ---
    if !controllerutil.ContainsFinalizer(db, myFinalizer) {
        controllerutil.AddFinalizer(db, myFinalizer)
        if err := r.Update(ctx, db); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{Requeue: true}, nil  // requeue to continue reconcile
    }

    // --- NORMAL RECONCILE PATH ---
    return r.reconcileDatabase(ctx, db)
}
🔴 Stuck terminating resources — emergency rescue If a controller crashes with a bug that prevents finalizer removal, objects get stuck in Terminating state forever. Emergency fix — force-patch the finalizers to empty:
kubectl patch database prod-postgres \
  --type=merge \
  -p '{"metadata":{"finalizers":[]}}'
This bypasses your cleanup logic. Only do this in emergencies after manually verifying external resources are cleaned up.

Checking for stuck resources

# Find all Terminating resources cluster-wide
kubectl get all -A | grep Terminating

# See which finalizers are blocking deletion
kubectl get database prod-postgres -o jsonpath='{.metadata.finalizers}'
# ["cleanup.myorg.example.com"]

# Check deletionTimestamp — if set, deletion is pending
kubectl get database prod-postgres -o jsonpath='{.metadata.deletionTimestamp}'
# 2024-01-15T10:30:00Z  ← deletion was requested at this time

# Check how long it's been stuck
kubectl get database prod-postgres
# NAME           PHASE               AGE
# prod-postgres  TerminationFailed   45m   ← stuck for 45 minutes

🗑️ Deletion Policies — Three Modes

When you delete a resource with dependents, Kubernetes supports three propagation policies. You control this via the --cascade flag or the propagationPolicy field in the delete options.

PolicyBehaviourOwner deleted when?Use case
Background (default) Owner is deleted immediately. GC controller deletes dependents asynchronously in the background. Immediately Normal deletion — fastest, most common
Foreground Owner gets foregroundDeletion finalizer. GC deletes all blockOwnerDeletion=true dependents first, then removes the finalizer, then deletes the owner. After all blocking dependents deleted When you need guaranteed cleanup ordering
Orphan Owner is deleted. Dependents' ownerReferences are cleared — they become independent objects (adopted by nobody). Immediately Keep child resources after parent gone (e.g. keep PVCs after StatefulSet delete)

Controlling deletion policy with kubectl

# Background (default) — owner gone immediately, GC cleans up children async
kubectl delete deployment my-app
kubectl delete deployment my-app --cascade=background   # explicit

# Foreground — wait for all dependents to be deleted first
kubectl delete deployment my-app --cascade=foreground

# Orphan — delete the Deployment but keep the ReplicaSets and Pods
kubectl delete deployment my-app --cascade=orphan

# Via API with propagationPolicy in delete options
kubectl proxy &
curl -X DELETE localhost:8001/apis/apps/v1/namespaces/default/deployments/my-app \
  -H "Content-Type: application/json" \
  -d '{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Foreground"}'

Foreground deletion — the lifecycle

kubectl delete --cascade=foreground API adds foregroundDeletion finalizer GC deletes dependents blockOwnerDeletion=true first Finalizer removed when all blocking dependents gone Owner deleted from etcd
⚠️ Foreground deletion and StatefulSets Using --cascade=foreground on a StatefulSet waits for all Pods to terminate before the StatefulSet is deleted. This can be slow for large StatefulSets. Use --cascade=orphan if you want to keep the Pods running (e.g. during a controller migration) or background for fast deletion.

Orphan policy — adopting orphaned resources

# Delete a ReplicaSet without deleting its Pods (orphan them)
kubectl delete rs my-app-7d9f8 --cascade=orphan

# The pods now have no controller — they won't be rescheduled if they die
# A new ReplicaSet with the same selector will "adopt" them if created

# Verify: pods still Running, no ownerReference
kubectl get pods -l app=my-app -o jsonpath='{range .items[*]}{.metadata.name}: {.metadata.ownerReferences}{"\n"}{end}'
# my-app-abc: []   ← orphaned, no owner

🧠 Knowledge Check

Q1. You delete a Deployment with --cascade=background. What is the state of the ReplicaSets and Pods immediately after the delete command returns?

A) All ReplicaSets and Pods are already deleted — cascade is synchronous
B) The Deployment is gone; ReplicaSets and Pods still exist temporarily — GC deletes them asynchronously
C) All resources enter Terminating state simultaneously
D> The Deployment enters Terminating and blocks until all Pods are gone

Q2. What is the difference between --cascade=foreground and --cascade=orphan?

A) Foreground is faster; orphan is slower but safer
B) They are identical for most resource types
C) Foreground waits for all blocking dependents to be deleted first; orphan keeps dependents alive as independent objects
D) Foreground only works for Deployments; orphan works for all resource types

Q3. A PVC is stuck in Terminating. kubectl get pvc my-pvc -o jsonpath='{.metadata.finalizers}' returns ["kubernetes.io/pvc-protection"]. What does this mean?

A) The PVC has a bug and needs to be force-patched
B) The storage provisioner failed to deprovision the underlying volume
C) A Pod is actively using the PVC — the protection finalizer blocks deletion to prevent data loss; delete the Pod first
D) The PVC has a reclaim policy of Retain and cannot be auto-deleted

Q4. Why can't ownerReferences cross namespace boundaries for namespaced resources?

A) It is a performance optimisation — cross-namespace GC is too slow
B) Namespaced resources use different API groups than cluster-scoped resources
C) It would violate namespace isolation — GC needs to look up the owner, which requires cross-namespace access
D) It is a planned feature that will be supported in a future Kubernetes version