guqiong96/Lvllm

LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, supporting hybrid inference for MOE large models.

What it solves

LvLLM addresses the memory limitations of GPUs when running large Mixture-of-Experts (MoE) models. It allows these models to run by splitting the model footprint across both GPU VRAM and system RAM, enabling the execution of models that would otherwise be too large for the available GPU memory.

How it works

LvLLM is an extension of vLLM that integrates the lk_moe engine to implement a hybrid GPU + NUMA (Non-Uniform Memory Access) parallel architecture. It allows MoE layers to flexibly switch between standard GPU computation and a hybrid CPU-GPU path. The system optimizes performance through NUMA-aware scheduling, thread binding to CPU cores, and a specialized GPU prefill mechanism that runs in parallel with CPU-GPU hybrid decoding to maximize hardware utilization.

Who it’s for

It is designed for users running large MoE models (such as Qwen3, GLM, and MiniMax series) on hardware where GPU VRAM is insufficient to hold the entire model, specifically those using x86 CPUs with AVX2+ and NVIDIA GPUs (SM75+).

Highlights

  • Dual Parallelism: Supports hybrid CPU-GPU decoding and prefill modes.
  • Memory Load Balancing: Combines VRAM and system memory to accommodate larger models.
  • NUMA Optimization: Reduces cross-node communication and improves L3 cache hit rates via specialized thread binding.
  • Broad Quantization Support: Compatible with bfloat16, float16, FP8, NVFP4, MXFP4, and AWQ 4-bit symmetric quantization.
  • vLLM Compatibility: Maintains full compatibility with the latest vLLM source code.