Project-HAMi/HAMi

What it solves

AI infrastructure teams often struggle with inefficient GPU utilization in Kubernetes, where whole GPUs are allocated to small jobs, creating scarcity and resource contention. HAMi addresses this by allowing multiple workloads to share a single physical accelerator, providing fine-grained resource isolation and device-aware scheduling across different hardware vendors.

How it works

HAMi acts as a Kubernetes-native virtualization middleware. It uses a mutating webhook, a scheduler extender, and device plugins to intercept pod submissions and allocate specific fractions of accelerator memory or compute cores. This allows workloads to request specific amounts of GPU memory (e.g., 3 GiB) without changing their application code, while HAMi coordinates the scheduling and isolation of these resources on the physical hardware.

Who it’s for

Platform engineers and AI infrastructure teams managing Kubernetes clusters that run multi-tenant AI workloads, such as notebooks, training, and inference servers, across heterogeneous accelerator pools (NVIDIA, Ascend, Cambricon, etc.).

Highlights

• Device Sharing: Allocate fractions of a physical accelerator by memory, core, or device count.
• Heterogeneous Support: Manages multiple accelerator types (GPUs, NPUs, HCUs, MLUs) through a single workflow.
• Device-Aware Scheduling: Supports binpack, spread, and topology-aware scheduling policies to optimize placement.
• Zero Application Changes: Works with standard Kubernetes resource requests and limits.
• Dynamic MIG: Dynamically creates, caches, and reclaims NVIDIA Multi-Instance GPU (MIG) instances.
• Ecosystem Integration: Integrates with vLLM for memory-capped inference and Volcano for batch scheduling.