kai-scheduler/KAI-Scheduler

What it solves

KAI Scheduler addresses the challenge of efficiently managing and allocating GPU resources in large-scale Kubernetes clusters. It prevents resource fragmentation and ensures fair distribution of compute power across different teams and workloads, ranging from small interactive jobs to massive distributed training and inference tasks.

How it works

Built on top of kube-batch, KAI Scheduler acts as a specialized Kubernetes scheduler that dynamically allocates GPUs. It uses advanced placement strategies like bin-packing to minimize fragmentation and spread scheduling for resiliency. It implements fairness policies such as Dominant Resource Fairness (DRF) and time-based fairshare to ensure no single user or team monopolizes the cluster.

Who it’s for

It is designed for administrators of large GPU clusters (potentially thousands of nodes) who need to orchestrate AI/ML workloads, including those using Ray or disaggregated serving architectures.

Highlights

• Topology-Aware Scheduling (TAS): Optimizes pod placement based on the physical hardware topology to improve performance.
• Hierarchical Queues: Allows organizational control via multi-level queues with specific quotas and priorities.
• Gang Scheduling: Ensures all pods in a group are scheduled simultaneously or not at all, preventing deadlocks.
• GPU Sharing: Enables multiple workloads to share single or multiple GPUs to maximize utilization.
• Dynamic Resource Allocation (DRA): Supports vendor-specific hardware like NVIDIA GB200/GB300.
• Elastic Workloads: Supports dynamic scaling of workloads within defined thresholds.