18 Batch processing with Jobs and CronJobs

This chapter explains how Kubernetes handles finite, run-to-completion workloads. While Deployments, StatefulSets, and DaemonSets keep Pods running indefinitely, many tasks need to run once or on a schedule. For one-off or bounded batches you use Jobs, and for time-based or recurring execution you wrap a Job in a CronJob. The chapter walks through creating, observing, suspending, and cleaning up these resources, and shows practical patterns for data initialization, batch simulations, aggregation tasks, and scheduled maintenance.

Jobs ensure a task completes successfully by creating one or more Pods and tracking their completion. You control scale and cadence with completions and parallelism, and decide where failures are handled via the Pod’s restartPolicy (OnFailure for in-Pod retries, Never for new Pods) along with backoffLimit, activeDeadlineSeconds, and optional podFailurePolicy rules. Jobs can start suspended, be suspended mid-run, auto-expire via ttlSecondsAfterFinished, and be deleted with or without cascading to their Pods. For input partitioning, NonIndexed Jobs treat Pods identically, whereas Indexed Jobs assign each Pod a stable index (exposed via JOB_COMPLETION_INDEX and an annotation) so each Pod processes a distinct slice; this enables fan-out processing and advanced schemes like using an init container to map indexes to concrete inputs. The chapter also covers queue-driven batches (coarse: one item per Pod; fine: Pods loop until the queue is empty), inter-Pod communication for indexed Jobs using a headless Service plus subdomain DNS, and the right way to run sidecars in Job Pods (as native sidecars via initContainers) so the Job can finish cleanly.

CronJobs add time-based orchestration: you define a Job template and a crontab-style schedule (including special shorthands and an optional timeZone). The controller creates Jobs on schedule, supports suspend/resume, and maintains history limits for successful and failed runs. Operational safeguards include startingDeadlineSeconds to avoid late or bursty catch-up runs and well-defined behavior when the controller has been offline. Concurrency is configurable: Allow (default), Forbid (skip if a prior run is still active), or Replace (terminate the previous run). The chapter also shows how to inspect status and events, label templates to discover created Jobs and Pods, and how deletion behaves—with either normal cascading (removing child Jobs/Pods) or orphaning to preserve them.

Three different Job examples. Each Job is completed once its pods have completed successfully.

An overview of the quiz-init Job

Running a parallel Job with completion = 5 and parallelism = 2

How failures are handled depending on the pod’s restart policy

Setting a time limit for a Job

Jobs using the NonIndexed completionMode spawn identical pods

Creating similar Jobs from a template

Pods spawned by a Job with the Indexed completion mode each get their own index number.

An init container providing the input file to the main container based on the job completion index

The difference between coarse and fine parallel processing

The operation of a CronJob

Comparing the behavior of the three CronJob concurrency policies

• A Job object is used to run workloads that execute a task to completion instead of running indefinitely.
• Running a task with the Job object ensures that the pod running the task is rescheduled in the event of a node failure.
• A Job can be configured to repeat the same task several times if you set the completions field. You can specify the number of tasks that are executed in parallel using the parallelism field.
• When a container running a task fails, the failure is handled either at the pod level by the Kubelet or at the Job level by the Job controller.
• By default, the pods created by a Job are identical unless you set the Job's completionMode to Indexed. In that case, each pod gets its own completion index. This index allows each pod to process only a certain portion of the data.
• You can use a work queue in a Job, but you must provide your own queue and implement work item retrieval in your container.
• Pods running in a Job can communicate with each other, but you need to define a headless Service so they can find each other via DNS.
• If a Job Pod requires a sidecar that never completes on its own, the sidecar must be defined as a native sidecar container (init container with a restartPolicy of Always).
• If you want to run a Job at a specific time or at regular intervals, you wrap it in a CronJob. In the CronJob, you define the schedule in the well-known crontab format.

FAQ