♻️ Idempotency — Run It a Million Times

A reconcile function must be idempotent: running it once or a thousand times with the same desired state must produce the same outcome with no side effects. The controller has no idea how many times it's been called — it must always be safe to call again.

Idempotent resource management pattern

// createOrUpdate — the idempotent resource management primitive
func (r *DatabaseReconciler) reconcileService(
    ctx context.Context,
    db *myorgv1.Database,
) error {
    desired := &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      db.Name + "-svc",
            Namespace: db.Namespace,
        },
        Spec: corev1.ServiceSpec{
            Selector: map[string]string{"app": db.Name},
            Ports: []corev1.ServicePort{{
                Port:       5432,
                TargetPort: intstr.FromInt(5432),
            }},
        },
    }

    // Set the Database as the owner so it's GC'd when the CR is deleted
    ctrl.SetControllerReference(db, desired, r.Scheme)

    // CreateOrUpdate: safe to call on every reconcile
    _, err := controllerutil.CreateOrUpdate(ctx, r.Client, desired,
        func() error {
            desired.Spec.Ports = []corev1.ServicePort{{
                Port:       5432,
                TargetPort: intstr.FromInt(5432),
            }}
            return nil
        })
    return err
}

Non-idempotent operations — handle carefully

Some operations are inherently non-idempotent (sending an email, charging a card, triggering a backup). Handle these with:

• Conditions in status — record "BackupInitiated" with a timestamp; skip if already set
• Annotations as semaphores — last-backup: "2024-01-15T10:00:00Z"
• Generation tracking — only act if db.Generation != db.Status.ObservedGeneration

// Skip backup if already done for this generation
if db.Status.ObservedGeneration == db.Generation &&
   db.Status.LastBackupTime != nil {
    return ctrl.Result{}, nil
}

// Trigger backup exactly once per generation change
if err := r.triggerBackup(ctx, db); err != nil {
    return ctrl.Result{}, err
}
db.Status.ObservedGeneration = db.Generation
db.Status.LastBackupTime = &metav1.Time{Time: time.Now()}
return ctrl.Result{}, r.Status().Update(ctx, db)

🌍 Real-World Operator Examples

Operator maturity levels

The Operator Framework defines five capability levels — a useful way to scope your operator's ambition:

⚡ Level-Triggered vs Edge-Triggered

This is one of the most important design concepts in Kubernetes controllers — and the source of many operator bugs when misunderstood.

The golden rule: never trust the event, always re-read

A watch event tells you "something changed". It does NOT tell you exactly what changed, and the event payload may be stale by the time you process it. The correct pattern:

// ❌ WRONG — edge-triggered: using the event payload directly
func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    event := getEventFromQueue()          // stale! another update may have happened
    if event.Type == "ADDED" {
        r.createStatefulSet(event.Object) // using potentially stale data
    }
    return ctrl.Result{}, nil
}

// ✅ CORRECT — level-triggered: always re-read current state
func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    db := &myorgv1.Database{}
    r.Get(ctx, req.NamespacedName, db)    // fresh read from API server cache

    // Work from current desired state, not from event payload
    desired := r.buildStatefulSet(db)
    existing := &appsv1.StatefulSet{}

    err := r.Get(ctx, client.ObjectKeyFromObject(desired), existing)
    if apierrors.IsNotFound(err) {
        return ctrl.Result{}, r.Create(ctx, desired)
    }
    // Compare and patch if different
    if !reflect.DeepEqual(existing.Spec, desired.Spec) {
        existing.Spec = desired.Spec
        return ctrl.Result{}, r.Update(ctx, existing)
    }
    return ctrl.Result{}, nil
}

Periodic resync — the safety net

Even with level-triggered reconciliation, watches can miss events (network partition, controller restart). controller-runtime's resync period re-enqueues every watched object periodically as a safety net:

// Set resync period when building the controller manager
mgr, _ := ctrl.NewManager(cfg, ctrl.Options{
    SyncPeriod: func() *time.Duration {
        d := 10 * time.Minute
        return &d
    }(),
})

// Or per-controller via WithOptions
ctrl.NewControllerManagedBy(mgr).
    For(&myorgv1.Database{}).
    WithOptions(controller.Options{
        MaxConcurrentReconciles: 5,
    }).
    Complete(r)

🧠 Knowledge Check