kubernetes-sigs/jobset

What it solves

JobSet provides a Kubernetes-native API to manage groups of Kubernetes Jobs as a single unit. It simplifies the deployment of distributed AI/ML training workloads (such as PyTorch, Jax, and TensorFlow) and HPC applications (like MPI) which typically require coordinated coordination between multiple pods, stable network endpoints, and specific failure recovery patterns.

How it works

JobSet models a distributed workload as a group of Kubernetes Jobs. It automates the management of headless services to provide stable pod hostnames for worker-to-worker communication. It also allows users to define multiple pod templates for different roles (e.g., leader and workers) and manages the lifecycle of the entire group based on configurable success and failure policies.

Who it’s for

Developers and platform engineers deploying large-scale distributed AI/ML training or HPC workloads on Kubernetes clusters.

Highlights

• Multi-template support: Define different pod templates for distinct groups of pods, such as leaders and parameter servers.
• Stable Networking: Automatic configuration of headless services and IndexedJobs to ensure stable pod hostnames for distributed training.
• Configurable Policies: Custom failure policies for automatic restarts from checkpoints and success policies to release resources efficiently.
• Exclusive Placement: Ensures a 1:1 mapping between a child job and a topology domain (like a rack or zone) for exclusive resource access.
• Startup Sequencing: Support for ordered startup of pods, enabling patterns like the leader-worker paradigm.
• Kueue Integration: Integration with Kueue for cluster oversubscription, multi-tenancy, and resource sharing.