kubernetes-sigs/lws

What it solves

This project provides Kubernetes APIs to manage the deployment of AI/ML inference workloads that require multiple pods to work together as a single unit. It specifically addresses the challenge of deploying large language models (LLMs) that are sharded across multiple devices and nodes, ensuring that these pods are created, scaled, and updated together rather than as independent entities.

How it works

The project introduces two primary APIs:

• LeaderWorkerSet (LWS): Manages a "super pod" consisting of a single leader pod and multiple worker pods. It ensures pods in a group have unique identities (indices), are created in parallel, and can be scheduled using gang scheduling (all-or-nothing).
• DisaggregatedSet (DS): Builds upon LWS to support advanced disaggregated architectures where different phases of inference (such as prefill and decode) run on separate infrastructure. It coordinates rollouts across multiple roles and manages headless services for discovery and routing.

Who it’s for

Infrastructure engineers and Kubernetes operators managing high-performance distributed inference serving stacks for large-scale AI models.

Highlights

• Unit-based Replication: Treats a group of pods as a single unit for scaling, rolling updates, and failure handling.
• Topology-aware Placement: Supports co-locating pods within the same group to optimize performance.
• All-or-nothing Restart: Ensures that if one pod in a group fails, the entire group is recreated to maintain consistency.
• Disaggregated Architecture Support: Specifically designed for separate prefill and decode phases in LLM inference.
• Coordinated Rollouts: Updates multiple roles in lockstep to preserve capacity ratios during updates.