Not every workload runs forever. Batch processing, migrations, reports, backups — these run to completion and then stop. Jobs and CronJobs are Kubernetes' answer to run-to-completion and scheduled workloads.

1. Jobs — Run to Completion

A Job creates one or more Pods and ensures they run successfully to completion (exit code 0). Unlike Deployments (which restart Pods forever), Jobs have a finite goal.

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migration
spec:
  completions: 1              # How many successful completions needed
  parallelism: 1              # How many Pods run simultaneously
  backoffLimit: 3             # Max retries before marking failed
  activeDeadlineSeconds: 300  # Max total runtime (kills Pod if exceeded)
  ttlSecondsAfterFinished: 600  # Auto-delete Job 10min after completion
  template:
    spec:
      restartPolicy: Never    # ← REQUIRED: Never or OnFailure
      containers:
        - name: migrate
          image: myapp:latest
          command: ["./migrate", "--up"]

Key Fields

FieldPurposeDefault
completionsTotal successful Pod completions needed1
parallelismMax Pods running at the same time1
backoffLimitMax retries before Job is marked Failed6
activeDeadlineSecondsHard time limit for entire Job (kills running Pods)None
ttlSecondsAfterFinishedAuto-cleanup completed/failed JobsNone (kept forever)

restartPolicy Requirement

PolicyOn FailureEffect
NeverPod stays Failed, Job creates a new PodYou can see logs of all attempts
OnFailureContainer is restarted in same PodCleaner (fewer Pods), but loses logs of previous attempts
Always❌ Invalid for JobsWould run forever — contradicts run-to-completion
Never vs OnFailure trade-off: Use Never when you need to inspect logs from failed attempts (each retry is a new Pod with preserved logs). Use OnFailure when failures are transient and you don't need per-attempt debugging — it keeps the namespace cleaner.

2. Parallelism Patterns

The combination of completions and parallelism creates different execution patterns:

Pattern 1: Single Completion (default)

completions: 1
parallelism: 1
# One Pod runs, must succeed once. Simplest case.
# Use: migrations, one-off scripts, backups

Pattern 2: Fixed Completion Count

completions: 10
parallelism: 3
# 10 items to process, 3 at a time
# Job creates Pods until 10 have succeeded
# Use: processing a known list of items (encode 10 videos)
completions=10, parallelism=3 t=0 Pod 1 Pod 2 Pod 3 t=1 Pod 4 Pod 2 Pod 5 ... continues until 10 Pods succeed ... t=end Pod 10 ✓ Job Complete

Pattern 3: Work Queue (Indexed Jobs)

completions: 10
parallelism: 3
completionMode: Indexed   # Each Pod gets JOB_COMPLETION_INDEX env var
# Pod 0 gets index=0, Pod 1 gets index=1, etc.
# Each Pod processes its assigned chunk (e.g., rows 0-999, 1000-1999, ...)
# Use: parallel data processing with partitioned work

Pattern 4: Work Queue (External Queue)

completions: null         # ← unset! (no fixed count)
parallelism: 5
# Pods pull work from an external queue (SQS, RabbitMQ, Redis)
# Job completes when a Pod exits successfully with no more work
# Use: variable-length work queues
Indexed Jobs (K8s 1.21+) assign each Pod a unique index via the JOB_COMPLETION_INDEX environment variable. This eliminates the need for an external queue for embarrassingly parallel workloads — each Pod knows exactly which chunk to process.

3. Failure Handling

backoffLimit

Each time a Pod fails (exits non-zero with restartPolicy: Never, or container restart count exceeds limit with OnFailure), it counts toward the backoff limit.

# backoffLimit: 3 means:
# Attempt 1: fails → retry
# Attempt 2: fails → retry  
# Attempt 3: fails → retry
# Attempt 4: fails → Job marked Failed (exceeded limit)

# Backoff timing between retries: 10s, 20s, 40s... (exponential, max 6min)

activeDeadlineSeconds

A hard wall-clock time limit. Once exceeded, all running Pods are terminated and the Job is marked Failed — regardless of how many completions have succeeded.

spec:
  activeDeadlineSeconds: 600   # Kill everything after 10 minutes
  # Use for: jobs that must finish within SLA, prevent runaway Pods
Always set activeDeadlineSeconds on production Jobs. A bug causing a Pod to hang indefinitely will consume resources forever if there's no deadline. Common pattern: set it to 2-3x the expected runtime.

Pod Failure Policy (K8s 1.26+)

Fine-grained control over which failures count toward backoffLimit:

spec:
  podFailurePolicy:
    rules:
      - action: Ignore            # Don't count this failure
        onPodConditions:
          - type: DisruptionTarget  # Node preemption — not our fault
      - action: FailJob            # Immediately fail the entire Job
        onExitCodes:
          containerName: migrate
          operator: In
          values: [42]             # Exit code 42 = unrecoverable error
      - action: Count              # Default: count toward backoffLimit
        onExitCodes:
          operator: NotIn
          values: [0]
Pod Failure Policy solves a real problem: Without it, a node preemption (not your code's fault) counts as a failure toward backoffLimit. With it, you can ignore infrastructure failures and only count application failures. FailJob lets you fast-fail on known-unrecoverable errors.

4. CronJobs — Scheduled Jobs

A CronJob creates Jobs on a time-based schedule (cron syntax). Think of it as crontab for Kubernetes.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"              # 2:00 AM daily
  timeZone: "America/New_York"       # (K8s 1.27+) explicit timezone
  concurrencyPolicy: Forbid          # Don't overlap runs
  startingDeadlineSeconds: 200       # Skip if missed by >200s
  successfulJobsHistoryLimit: 3      # Keep last 3 successful Jobs
  failedJobsHistoryLimit: 1          # Keep last 1 failed Job
  suspend: false                     # Set true to pause scheduling
  jobTemplate:
    spec:
      activeDeadlineSeconds: 3600
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: backup-tool:latest
              command: ["./backup.sh"]

Cron Schedule Syntax

┌───────────── minute (0-59)
│ ┌───────────── hour (0-23)
│ │ ┌───────────── day of month (1-31)
│ │ │ ┌───────────── month (1-12)
│ │ │ │ ┌───────────── day of week (0-6, Sun=0)
│ │ │ │ │
* * * * *

Examples:
"*/5 * * * *"     Every 5 minutes
"0 * * * *"       Every hour
"0 2 * * *"       Daily at 2AM
"0 2 * * 1"       Every Monday at 2AM
"0 0 1 * *"       First day of month at midnight

concurrencyPolicy

PolicyBehaviorUse Case
Allow (default)Multiple Jobs can run concurrentlyIndependent tasks (each run processes different data)
ForbidSkip new Job if previous is still runningTasks that must not overlap (backup, migration)
ReplaceKill the running Job, start a new oneLatest data matters more than completing old run
For database backups, always use Forbid. If a backup takes longer than expected and overlaps the next scheduled run, you don't want two backup processes competing for locks. The missed run is skipped, and you get alerted via monitoring.

startingDeadlineSeconds

If the CronJob controller misses a scheduled time (e.g., controller was down), it counts missed schedules. If the miss exceeds startingDeadlineSeconds, it skips that run entirely.

# If startingDeadlineSeconds: 200
# Schedule: every hour at :00
# Controller was down from 2:00 to 2:05
# At 2:05, it sees it missed the 2:00 run (5min ago < 200s deadline)
# → Creates the Job (late but within deadline)

# If controller was down from 2:00 to 3:00
# At 3:00, the 2:00 run was missed by 3600s > 200s
# → Skips the 2:00 run, only creates the 3:00 run
If >100 missed schedules accumulate (e.g., CronJob was suspended for days then unsuspended), the CronJob controller refuses to create any Job and logs an error. This prevents a "thundering herd" of catch-up Jobs.

5. Cleanup & History

TTL After Finished (Jobs)

spec:
  ttlSecondsAfterFinished: 3600   # Delete Job + Pods 1 hour after completion
# Without this, finished Jobs and their Pods linger forever
# Clutters etcd and namespace

History Limits (CronJobs)

spec:
  successfulJobsHistoryLimit: 3    # Keep last 3 successful Job objects
  failedJobsHistoryLimit: 1        # Keep last 1 failed Job object
# Older Jobs (and their Pods) are automatically deleted
# Keeps namespace clean while allowing debugging of recent failures
In clusters with many CronJobs, uncleaned Job objects accumulate in etcd and can impact API server performance. Always set either ttlSecondsAfterFinished (on Jobs) or history limits (on CronJobs). A common pattern: successfulJobsHistoryLimit: 3, failedJobsHistoryLimit: 5 (keep more failures for debugging).

Manual Job Trigger

# Create a Job from a CronJob (run immediately, outside schedule):
kubectl create job backup-manual --from=cronjob/nightly-backup

# Useful for: testing, incident response, ad-hoc runs
kubectl create job --from=cronjob/name is tested in CKAD. It creates a one-off Job using the CronJob's template. The Job runs immediately regardless of the schedule.

Summary

ConceptKey Point
JobRun Pods to completion (exit 0), retries on failure
completionsHow many successful Pods needed (total work items)
parallelismHow many Pods run simultaneously
backoffLimitMax retries before marking Job as Failed
activeDeadlineSecondsHard time limit — kills everything if exceeded
Indexed JobsEach Pod gets JOB_COMPLETION_INDEX — parallel processing without external queue
CronJobCreates Jobs on a cron schedule
concurrencyPolicyAllow / Forbid / Replace — controls overlap
startingDeadlineSecondsHow late a missed schedule can still be triggered
restartPolicyMust be Never or OnFailure (not Always)

📝 Quiz: Jobs & CronJobs

Q1: A Job has completions: 5, parallelism: 2, backoffLimit: 3. Pod 1 and 2 start. Pod 1 succeeds, Pod 2 fails. What happens next?

The failure counts toward the backoff limit (now 1/3 used). The Job creates two new Pods (maintaining parallelism=2): one replacement for the failed Pod, and the next Pod in sequence. The Job continues until 5 Pods succeed or backoffLimit is exceeded.

Q2: What's the difference between restartPolicy: Never and restartPolicy: OnFailure for a Job?

Never: On failure, the Pod stays in Failed state (you can read its logs). The Job controller creates a new Pod for the retry. Multiple failed Pods may accumulate.
OnFailure: On failure, the kubelet restarts the container in the same Pod. The Pod stays, but container restart count increases. Cleaner namespace but loses per-attempt logs.

Q3: A CronJob runs every hour. The current Job takes 90 minutes. concurrencyPolicy: Forbid is set. What happens at the next scheduled time?

The next scheduled run is skipped. With Forbid, if the previous Job is still running when the next schedule triggers, the new Job is not created. Once the current Job finishes, the following scheduled time will create a new Job normally.

Q4: You have an Indexed Job with completions: 100, parallelism: 10. How does Pod #47 know what work to do?

It reads the environment variable JOB_COMPLETION_INDEX=47. The application uses this index to determine its chunk of work (e.g., process database rows 47000-47999, or file shard 47). No external coordination needed.

Q5: A Job has activeDeadlineSeconds: 300. After 300 seconds, 4 of 5 completions have succeeded. What happens?

The Job is marked Failed and all running Pods are terminated. The activeDeadlineSeconds is a hard wall-clock limit that overrides everything else — it doesn't matter that 4/5 were already done. The partial progress is lost (for that Job instance).

Q6: How do you manually trigger a CronJob to run immediately (outside its schedule)?

kubectl create job my-manual-run --from=cronjob/my-cronjob
This creates a one-off Job using the CronJob's jobTemplate. It runs immediately and independently of the schedule.