Project-HAMi/HAMi-core

A transparent, in-container GPU resource controller that enforces memory and compute limits by intercepting CUDA calls without application or driver changes.

What it solves

HAMi-core is an in-container GPU resource controller that allows users to enforce strict limits on device memory and compute utilization (SM utility) for CUDA applications. It solves the problem of GPU resource fragmentation and inefficient allocation by enabling virtualization of GPU resources within containers, preventing a single application from consuming all available GPU memory or compute power.

How it works

The library operates by hijacking (intercepting) the API calls between the CUDA Runtime (libcudart.so) and the CUDA Driver (libcuda.so). By sitting in the middle of these calls, HAMi-core can monitor and enforce memory and compute limits without requiring any changes to the application code or the NVIDIA driver.

Who it’s for

This tool is designed for developers and platform engineers managing GPU-accelerated containers, particularly those using orchestrators like Volcano or the broader HAMi project to manage shared GPU resources.

Highlights

  • Zero Application Changes: Enforces limits without modifying the source code of the CUDA application.
  • Memory Virtualization: Virtualizes device memory to set specific upper limits per container.
  • Compute Limiting: Limits device utilization via a self-implemented time-shard mechanism.
  • Real-time Monitoring: Includes a built-in monitor for device utilization.

Related

  • Project
    Project-HAMi/HAMi

    A Kubernetes GPU virtualization and scheduling middleware that enables sharing and isolating accelerator resources across multiple AI workloads to increase hardware utilization.

  • Project
    godweiyang/GrabGPU

    A CUDA-based utility that occupies GPU memory and compute resources to simulate active usage and prevent resource reclamation in shared clusters.

  • Project
    rapidsai/rmm

    RMM (RAPIDS Memory Manager) is a C++/Python library that provides a unified interface for GPU memory allocation. It offers multiple allocation strategies (plain CUDA, pooled, fixed‑size, managed, etc.), stream‑ordered allocation, and per‑device default resources. Users can install via pip or conda, or build from source, and then use the API to set custom allocators globally or per‑allocation, improving performance of GPU‑centric workloads.

  • Project
    llnl/Umpire

    Umpire is a resource management library for discovering and managing memory across machines with multiple memory devices like NUMA and GPUs.

  • Project
    eyalroz/cuda-api-wrappers

    A header-only C++ library providing thin, modern wrappers for NVIDIA's CUDA Runtime, Driver, NVRTC, and NVTX APIs to simplify GPU programming and resource management.