Kubernetes Jobs and CronJobs Explained (2026): backoffLimit to timeZone

I have watched more nightly backups silently stop running than I have watched clusters fall over, and almost every time the cause was one field in a CronJob or Job spec that nobody had read the docs for. Kubernetes Jobs and CronJobs look like the simplest objects in the API: run a container to completion, maybe on a schedule. They are not simple. This post explains how Kubernetes Jobs and CronJobs actually behave in v1.37 (the current release is v1.37.2, published 28 October 2026): the three Job shapes, how retries are counted, the newer failure and success policies that went GA between 1.31 and 1.35, how cleanup works, and how the CronJob controller decides whether to start a run. At the end I cover when a CronJob is the wrong tool and Argo Workflows or Kueue fits better.

What is a Kubernetes Job, and how is it different from a Deployment?

A Deployment keeps N Pods running forever and replaces any that exit. A Job does the opposite: it creates Pods until a certain number of them exit successfully, then stops. The Job documentation defines it as a controller that tracks successful completions, and that definition matters, because "success" is the only thing the Job controller really cares about.

Two consequences follow. First, restartPolicy must be Never or OnFailure, never Always. With OnFailure the kubelet restarts the failed container inside the same Pod; with Never the Job controller creates a fresh Pod. Second, your program must tolerate being started more than once. The docs say plainly that even with parallelism: 1, completions: 1 and restartPolicy: Never, "the same program may sometimes be started twice". Node failures, evictions and kubelet restarts all cause this. If your Job sends emails or charges cards, idempotency is your job, not Kubernetes'.

A CronJob is a thin layer on top: one CronJob object is roughly one line of a crontab, and on each schedule tick it creates a Job from .spec.jobTemplate. The CronJob page notes that CronJob has been stable in batch/v1 since v1.21, and that it only creates Jobs; everything about Pods and retries belongs to the Job.

Completions, parallelism and completionMode: the three Job shapes

Almost every Job is one of three shapes, and the shape is set entirely by .spec.completions, .spec.parallelism and .spec.completionMode.

Shape	completions	parallelism	Done when	Typical use	Source
Single task	unset (defaults to 1)	unset (defaults to 1)	one Pod succeeds	migration, report, backup	Job docs: parallel execution
Fixed completion count	N	1 to N	N Pods succeed	process N shards	Job docs: parallel execution
Indexed (fixed count)	N, plus completionMode: Indexed	1 to N	one success per index 0 to N-1	static sharding, distributed training	Job docs: completion mode
Work queue	unset	M	any Pod succeeds and all have exited	pull items from Redis or SQS	Job docs: parallel execution

Indexed Jobs are the one I would learn first, because they remove the need for an external queue in most batch work. Each Pod gets an index through the JOB_COMPLETION_INDEX environment variable, the batch.kubernetes.io/job-completion-index annotation and label, and a stable hostname of the form $(job-name)-$(index). Indexed completion mode has been GA since v1.24. Pair it with a headless Service and the Pods can find each other by DNS, which is exactly what MPI and PyTorch-style leader/worker jobs need.

Setting parallelism: 0 pauses a Job, and parallelism above the remaining completions is simply ignored. Actual parallelism can also be lower than requested if a ResourceQuota blocks Pod creation or the controller is throttling after failures. If your Pods sit in Pending, that is a scheduling problem, not a Job problem; my Pod Pending guide covers it.

How Jobs handle failure: backoffLimit, podFailurePolicy and per-index retries

The default retry model is blunt. .spec.backoffLimit defaults to 6, and failed Pods are recreated with an exponential delay of 10s, 20s, 40s and so on, capped at ten minutes, per the pod backoff failure policy section. When the count of failed Pods (or, with OnFailure, container restarts) reaches the limit, the Job gets a Failed condition with reason BackoffLimitExceeded. A separate wall-clock cap, .spec.activeDeadlineSeconds, fails the Job with reason DeadlineExceeded and takes precedence over backoffLimit.

The problem with a single counter is that it treats every failure the same. A Pod preempted by a higher-priority workload and a Pod that hit a bug in your code both burn a retry. That is what podFailurePolicy fixes. It became GA in v1.31 (the feature gate table lists JobPodFailurePolicy as stable from 1.31). Rules are evaluated in order and match on container exit codes or Pod conditions, with four actions: FailJob, Ignore, Count and FailIndex. It requires restartPolicy: Never.

For Indexed Jobs there is a second counter, backoffLimitPerIndex, GA in v1.33. Each index gets its own retry budget, a failing index does not stop the others, and maxFailedIndexes aborts the whole Job once too many indexes have failed. When you set backoffLimitPerIndex, the global backoffLimit defaults to 2147483647 so it effectively stops mattering.

Here is a Job I would consider a sane default for sharded batch work on v1.33 or later:

# indexed-batch-job.yaml  (batch/v1, tested against the Kubernetes v1.37 API docs)
apiVersion: batch/v1
kind: Job
metadata:
  name: reindex-shards
spec:
  completions: 20              # 20 shards, indexes 0..19
  parallelism: 5               # at most 5 Pods at once
  completionMode: Indexed      # JOB_COMPLETION_INDEX is injected into each Pod
  backoffLimitPerIndex: 2      # each shard may fail twice (GA in v1.33)
  maxFailedIndexes: 3          # give up on the whole Job if 4+ shards fail
  activeDeadlineSeconds: 7200  # hard 2-hour cap, wins over any retry budget
  ttlSecondsAfterFinished: 86400 # delete Job and Pods one day after it finishes
  podFailurePolicy:            # GA in v1.31; needs restartPolicy: Never
    rules:
    - action: FailIndex        # exit 42 = bad input for this shard, do not retry it
      onExitCodes:
        containerName: worker
        operator: In
        values: [42]
    - action: Ignore           # preemption, eviction, node drain: retry for free
      onPodConditions:
      - type: DisruptionTarget
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: python:3.13-slim
        command: ["python", "-c"]
        args:
        - |
          import os
          shard = int(os.environ["JOB_COMPLETION_INDEX"])
          print(f"processing shard {shard}")
        resources:
          requests: {cpu: "250m", memory: "256Mi"}
          limits: {memory: "512Mi"}

Two related fields are worth knowing. podReplacementPolicy: Failed (GA in v1.34) tells the controller to wait until a terminating Pod is fully gone before creating its replacement, which stops two copies of the same shard running during a slow shutdown; it is already the default whenever a podFailurePolicy is set. And managedBy (GA in v1.35) hands reconciliation of a Job to an external controller, for example a multi-cluster dispatcher such as Kueue's MultiKueue, instead of the built-in one. Exit code 137 from the memory limit above is covered in my OOMKilled guide.

What is successPolicy, and when do you need it?

By default a Job succeeds only when every completion is accounted for. successPolicy, GA in v1.33 and only valid for Indexed Jobs, lets you declare success earlier. A rule can name specific indexes (succeededIndexes: "0"), a count (succeededCount: 5), or both. The success policy docs give the two motivating cases: parameter sweeps where you only need some runs to pass, and leader/worker frameworks where only the leader's exit code matters.

When a rule matches, the Job gets a SuccessCriteriaMet condition with reason SuccessPolicy, the controller terminates the lingering Pods, and then the Job becomes Complete. One subtlety: if both a success policy and a terminating policy (backoff limit or pod failure policy) are satisfied, the terminating policy wins. A failed Job that "also succeeded" is still failed.

Cleaning up: ttlSecondsAfterFinished and Job history limits

Finished Jobs and their Pods are not deleted by default. That is deliberate, so you can read logs, but on a busy cluster thousands of finished Pods add load to the API server and etcd. There are two cleanup mechanisms, and they apply to different Jobs.

For Jobs you create directly, set ttlSecondsAfterFinished. The TTL-after-finished controller, stable since v1.23, deletes the Job and its Pods cascadingly that many seconds after it reaches Complete or Failed; 0 means immediately. The Job docs recommend setting it explicitly, because a Job deleted without cascading options can leave orphaned Pods behind.

For Jobs a CronJob creates, the CronJob's successfulJobsHistoryLimit (default 3) and failedJobsHistoryLimit (default 1) keep the newest finished Jobs and delete the rest. Keep failedJobsHistoryLimit at 1 or higher. Setting it to 0 means the only evidence of last night's failure is an event, and events are kept for one hour by default (the kube-apiserver --event-ttl flag).

How does the CronJob controller schedule runs?

This is where most surprises live, because the CronJob controller is a loop that compares "now" with "when should the last run have been", not a precise timer. The fields that control it:

schedule and timeZone. The schedule is standard five-field cron (minute, hour, day of month, month, day of week) plus macros like @daily, parsed with robfig/cron. Without timeZone, the schedule is interpreted in the kube-controller-manager's local time zone, which you usually cannot see on a managed control plane. Set timeZone to an IANA name such as Europe/London; it has been GA since v1.27. Putting CRON_TZ= or TZ= inside the schedule string is rejected with a validation error.

concurrencyPolicy. Allow (the default) lets runs overlap. Forbid skips a run if the previous Job is still active, and records a JobAlreadyActive event. Replace deletes the running Job and starts the new one. Only Jobs from the same CronJob count.

startingDeadlineSeconds. How late a run may start. If the controller wakes up and the scheduled time plus this deadline is in the past, the run is skipped with a MissSchedule warning. The docs warn that values under 10 seconds may mean the run never starts, because the controller checks on a 10-second cycle.

The 100 missed schedules rule. After downtime, the controller counts schedules it missed since the last run (or within the last startingDeadlineSeconds, if set). In the controller source for v1.37.2, more than 100 triggers a TooManyMissedTimes warning. As of the docs revision of 31 October 2026, the page states clearly that it still starts one Job for the most recent missed time; earlier missed runs never happen. CronJobs do not backfill.

suspend. Pauses new runs without touching running Jobs. Unsuspending a CronJob that has no starting deadline immediately starts a Job for the most recent missed schedule, which surprises people who expected the next tick.

# nightly-backup-cronjob.yaml  (batch/v1, timeZone GA since v1.27)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup            # 52 characters max: the controller appends 11
spec:
  schedule: "30 2 * * *"          # 02:30 every day
  timeZone: "Europe/London"       # IANA name; never CRON_TZ= in the schedule
  concurrencyPolicy: Forbid       # never run two backups at once
  startingDeadlineSeconds: 3600   # a run up to 1 hour late is still useful
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3       # keep failures around long enough to debug
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 5400 # 90 minutes, then fail and alert
      template:
        spec:
          restartPolicy: Never    # fresh Pod per retry keeps logs of each attempt
          containers:
          - name: backup
            image: busybox:1.37
            command: ["sh", "-c", "date; echo backup would run here"]

Every Job a CronJob creates is named <cronjob-name>-<scheduled time in minutes since the epoch>, and since v1.28 (GA in v1.32) it carries a batch.kubernetes.io/cronjob-scheduled-timestamp annotation with the intended time. The deterministic name is why a restarted controller does not create the same run twice. Duplicates come from elsewhere, and my companion guide on fixing a CronJob that is not running or runs twice walks through each cause with commands.

One DST warning: if your timeZone observes daylight saving, schedules inside the skipped or repeated hour are risky. The Kubernetes docs say nothing about it, but Argo's CronWorkflow docs, which use the same robfig/cron parser, show such runs happening twice or not at all on changeover days. Either schedule outside 01:00 to 03:00 local time or use Etc/UTC.

When should you use Argo Workflows or Kueue instead of a CronJob?

A CronJob runs one Pod template on a timer. It has no dependencies between steps, no artifact passing, no retries per step, and no way to say "run B only after A succeeded". The moment you chain two CronJobs by time ("extract at 02:00, load at 03:00 and hope"), you have built a fragile workflow engine.

Argo Workflows (latest release v4.1.6, 22 October 2026) has a CronWorkflow that deliberately mimics CronJob's options: concurrencyPolicy, startingDeadlineSeconds, history limits, suspend and timezone. It adds a list of schedules, a when expression evaluated at each tick, and a DAG of steps. If you need data pipelines with Python-first authoring, Airflow, Prefect or Dagster on Kubernetes are the other choices; see ML pipeline orchestrators compared.

Kueue (v0.20.5 at the time of writing) solves a different problem: queueing and quota. It holds Jobs suspended until capacity is available, then admits them, which matters when many teams share GPUs. It works with plain batch/v1 Jobs through the suspend field and managedBy, so you do not have to rewrite anything. If GPU sharing is your real issue, start with GPUs on Kubernetes explained.

My rule of thumb: one step, idempotent, on a timer, use a CronJob. Two or more steps with dependencies, use a workflow engine. Many teams fighting over capacity, add Kueue in front of whichever you chose.

FAQ

What is the difference between a Kubernetes Job and a CronJob?

A Job runs Pods until a set number complete successfully, then stops. A CronJob creates a new Job on a cron schedule. Retries, parallelism and failure policies are all configured on the Job, inside the CronJob's jobTemplate.

What is the default backoffLimit for a Kubernetes Job?

Six retries. Failed Pods are recreated with exponential backoff starting at 10 seconds and capped at ten minutes. If you set backoffLimitPerIndex on an Indexed Job, the global backoffLimit defaults to the maximum 32-bit integer instead.

Should a Job use restartPolicy Never or OnFailure?

Use Never unless you have a reason not to. It is required for podFailurePolicy, and each retry gets a new Pod whose logs you can read. OnFailure restarts the container in place, which is faster but terminates the Pod when the limit is reached, taking the logs with it.

Does a Kubernetes CronJob run missed schedules?

Only one. After an outage or a suspend, the controller starts at most one Job for the most recent missed time, provided it is within startingDeadlineSeconds if you set one. It never replays a backlog, so design catch-up logic into the job itself if you need it.

This quarter I would grep every CronJob in our clusters for a missing timeZone, a failedJobsHistoryLimit of 0, and Jobs without activeDeadlineSeconds, then add podFailurePolicy with a DisruptionTarget ignore rule to anything that runs on spot nodes. None of that is glamorous. It is the difference between a backup that runs and a backup you believe runs.

Related: How to Fix Kubernetes CrashLoopBackOff: Causes and Commands (2026), Kubernetes Requests and Limits Explained: 8% CPU Used, Resize GA in 1.35 and Backup and Disaster Recovery in 2026: RPO vs RTO, 3-2-1-1-0, Real Costs.