kubernetes-sigs/lws

What it is

LeaderWorkerSet (LWS) and DisaggregatedSet (DS) are new Kubernetes APIs designed to make it easier to run large‑scale AI/ML inference workloads – especially those that need to be split (sharded) across many GPUs or other accelerators on multiple nodes.

LeaderWorkerSet lets you treat a group of pods as a single logical replica. Each group contains one leader pod and a set of worker pods that all share the same lifecycle. The group gets a unique index (0‑n) and can be scheduled together (gang scheduling) so the whole set lands on the same topology (e.g., same rack or node pool).
DisaggregatedSet builds on LWS to handle more complex, multi‑stage inference pipelines (e.g., a pre‑fill stage on one set of machines and a decode stage on another). It coordinates several LWS groups, keeping capacity ratios steady during roll‑outs and handling failures across all roles.

Both APIs are being co‑designed with the llm‑d project – a CNCF sandbox distributed inference serving stack – so they map closely to real‑world serving frameworks.

Why it matters for AI/ML

Running large language models (LLMs) in production often means:

Sharding the model across many GPUs or TPUs.
Co‑ordinating multiple phases (prefill, decode, post‑processing) that may run on different hardware.
Keeping the whole pipeline alive – if any pod crashes, the inference request can fail.

Standard Kubernetes Deployments treat each pod independently, which makes it hard to guarantee that all parts of a sharded model start, stop, and scale together. LWS gives you a super‑pod that behaves as a single unit, while DS lets you orchestrate several such super‑pods in lock‑step. This reduces operational complexity and improves reliability for high‑throughput, low‑latency inference services.

Core features (LeaderWorkerSet)

Feature	What it does
Group of pods as a unit	A replica is a leader + workers that share lifecycle; they are created in parallel and can be rolled out together.
Unique pod identity	Each pod gets an index (0‑n) useful for model sharding logic.
Gang scheduling (alpha)	All pods in a group are scheduled together or not at all, ensuring co‑location.
Dual‑template	Separate pod templates for the leader and the workers, allowing different resource specs.
Multiple groups	You can create many identical replicas; each group is a unit for scaling and updates.
Scale subresource	Exposes a `/scale` endpoint so the Horizontal Pod Autoscaler (HPA) can adjust the number of groups.
Rolling updates at group level	Updates happen group‑by‑group, keeping the whole set consistent during a rollout.
Topology‑aware placement	Optionally forces pods in a group to be placed on the same node, zone, or other topology.
All‑or‑nothing restart	If any pod in the group fails, the whole group is recreated, simplifying failure handling.

Advanced features (DisaggregatedSet)

Feature	What it does
Disaggregated architecture	Supports pipelines where different roles (e.g., pre‑fill, decode) run on separate hardware clusters.
Coordinated N‑dimensional rollouts	Updates multiple roles (2‑10) together while preserving the desired capacity ratios.
Unified lifecycle	Treats several LWS objects as one logical unit for scaling, updates, and deletion.
Automatic service orchestration	Creates headless services for each role so pods can discover each other without extra config.
Advanced failure handling	Coordinates draining and restarts across all roles to keep the overall pipeline healthy.

Who might use it

ML platform teams building production inference services for large LLMs that need to be sharded across many GPUs.
Cloud providers offering managed inference stacks that want tighter integration with Kubernetes.
Research groups that run multi‑node model serving experiments and need reproducible, versioned deployments.

Getting started

Install – Follow the installation guide (requires Kubernetes 1.34+). The project provides CRD manifests and a controller that runs inside the cluster.
Define a LeaderWorkerSet – Write a YAML that specifies the leader pod template, the worker pod template, and the desired replica count.
(Optional) Use DisaggregatedSet – If your workload has distinct roles, create a DS object that references multiple LWS objects.
Scale & monitor – Use the standard kubectl scale or HPA on the /scale subresource; the controller handles gang‑scheduling and coordinated roll‑outs.

Community & support

Slack: #sig-apps on the Kubernetes Slack workspace.
Mailing list: kubernetes-sig-apps Google Group.
Documentation & talks: https://lws.sigs.k8s.io/docs/ (includes examples, adoption stories, and presentation videos).
Contributing: Follow the usual Kubernetes SIG process – submit PRs, run the CI (unit, integration, and E2E tests across K8s 1.34‑1.37), and adhere to the Kubernetes Code of Conduct.

TL;DR

LWS + DS give Kubernetes a first‑class way to deploy and manage multi‑pod, multi‑node inference workloads (like sharded LLM serving). They provide group‑level lifecycle, gang scheduling, scaling, and coordinated roll‑outs, making large‑scale AI inference more reliable and easier to operate on Kubernetes.