EfficientMoE/MoE-Infinity

What it solves

MoE-Infinity addresses the high memory requirements of large Mixture-of-Experts (MoE) models, which often exceed the capacity of available GPU VRAM. It allows these models to run on memory-constrained GPUs by offloading expert weights to host memory (RAM) and SSDs, reducing the hardware cost required for inference.

How it works

The library uses an offloading runtime that moves expert weights from the GPU to the host memory and SSD, fetching them only when needed. To maintain performance, it employs an activation-aware cache to keep frequently used "hot" experts on the GPU, alongside tracing and prefetching to hide the latency of data transfers. It also supports FP4 and FP8 quantization for specific models to further reduce memory footprints. For serving, it provides an OpenAI-compatible engine featuring continuous batching, a paged KV cache, and streaming.

Who it’s for

It is designed for developers and researchers who want to run large MoE models (such as DeepSeek, Mixtral, or Qwen) on hardware with limited GPU memory, as well as those needing a production-ready serving engine for these models.

Highlights

• Cost-effective offloading: Moves expert weights to host RAM and SSD to fit massive models on smaller GPUs.
• Performance optimizations: Uses fused CUDA kernels, CUDA graph capture, and Marlin INT4 GEMM to keep the hot path efficient.
• HuggingFace integration: Provides a HuggingFace-compatible MoE class for easy model loading.
• Production-ready serving: Includes an OpenAI-compatible HTTP server with continuous batching and request scheduling.
• Multi-GPU support: Distributes expert parameters across multiple GPUs in a single server using round-robin distribution.
• Acceleration awareness: Automatically integrates with FlashAttention and FlashInfer for optimized attention kernels.