kai-scheduler/KAI-Scheduler

What it solves

KAI Scheduler addresses the challenge of efficiently allocating GPU resources in large-scale Kubernetes clusters. It prevents resource fragmentation and ensures fair access to expensive hardware for AI and machine learning workloads, ranging from small interactive jobs to massive distributed training and inference tasks.

How it works

Built on top of kube-batch, KAI Scheduler optimizes GPU placement using several advanced scheduling strategies:

• Resource Management: It uses Hierarchical Queues and Dominant Resource Fairness (DRF) to manage quotas and ensure equitable distribution across teams.
• Placement Optimization: It employs Bin Packing to minimize fragmentation and Topology-Aware Scheduling (TAS) to place workloads based on the physical layout of the hardware for better performance.
• Workload Handling: It supports "Gang Scheduling" (ensuring all pods in a group start together) and Elastic Workloads that can scale within defined thresholds.
• Hardware Integration: It integrates with Kubernetes Dynamic Resource Allocation (DRA) to support vendor-specific hardware like NVIDIA GB200/GB300 GPUs.

Who it’s for

It is designed for Kubernetes cluster administrators and ML engineers managing large GPU clusters (potentially thousands of nodes) that need to operate in both cloud and on-premise environments.

Highlights

• Topology-Aware Scheduling: Optimizes placement for disaggregated serving architectures.
• Time-based Fairshare: Considers historical usage to ensure long-term fairness.
• GPU Sharing: Allows multiple workloads to share GPUs to maximize utilization.
• Hierarchical PodGroups: Supports complex, multi-level gang scheduling for agentic pipelines.
• KubeRay Integration: Native support for Ray workloads on Kubernetes.