sgl-project/rbg

What it solves

Traditional Kubernetes primitives like StatefulSets and Deployments are not designed for the complex requirements of LLM inference services. These services often require multi-role topologies (such as separating prefill and decode roles), strict hardware affinity (GPU-NVLink, RDMA), and atomic operations across different roles during scaling or upgrades.

How it works

RoleBasedGroup (RBG) is a Kubernetes API that treats an entire inference service as a single, coordinated unit. It introduces several key concepts:

• Roles: The basic scheduling unit where each role (e.g., prefill or decode) has its own specification and lifecycle.
• RoleBasedGroup: A logical grouping of these roles that form one service.
• RoleInstance: A collection of Pods with a bound lifecycle, supporting in-place updates.
• CoordinatedPolicy: A separate CRD used to manage rolling updates and scaling across multiple roles simultaneously to maintain stability.

Who it’s for

It is designed for engineers and operators deploying distributed, stateful AI inference workloads on Kubernetes, particularly those using engines like SGLang or vLLM.

Highlights

• Topology-Awareness: Deterministic operations with unique RoleID injection.
• Cross-Role Coordination: A policy engine for paired deployment, linked recovery, and coordinated upgrades.
• Hardware Affinity: Scheduling optimized for GPU-NVLink, PCIe, RDMA, and VPC.
• Service Discovery: Built-in, topology-aware service discovery for distributed roles.
• Ecosystem Integration: Native support for NVIDIA Dynamo and Mooncake for KV cache reuse.