kubernetes-sigs/jobset

What it solves

JobSet provides a Kubernetes-native way to manage groups of Jobs as a single unit. It solves the difficulty of deploying distributed AI/ML training workloads (like PyTorch, Jax, and TensorFlow) and HPC applications (like MPI) on Kubernetes, which typically require stable network endpoints, specific pod templates for different roles, and coordinated failure recovery.

How it works

JobSet acts as a higher-level API that orchestrates multiple Kubernetes Jobs. It manages the lifecycle of these jobs, ensuring that pods have stable hostnames via IndexedJobs and automatic headless service configuration for pod-to-pod communication. It allows users to define different pod templates for different roles (e.g., leader vs. worker) and controls the startup sequence and success/failure policies across the entire group.

Who it’s for

It is designed for developers and platform engineers deploying large-scale distributed training or high-performance computing workloads on Kubernetes clusters.

Highlights

• Multi-template support: Define distinct pod templates for different groups of pods, such as leaders, workers, and parameter servers.
• Stable Networking: Automatically configures headless services to provide stable pod hostnames required by ML and HPC frameworks.
• Customizable Policies: Set specific success and failure policies to determine when a workload is complete or when the entire set should be restarted from a checkpoint.
• Exclusive Placement: Ensures a 1:1 mapping between a child job and a topology domain (like a rack or zone) to give jobs exclusive access to local resources.
• Startup Sequencing: Allows configuring the order in which ReplicatedJobs start, supporting leader-worker patterns.
• Kueue Integration: Integrates with Kueue for cluster oversubscription, multi-tenancy, and resource sharing.