guqiong96/Lvllm

LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, supporting hybrid inference for MOE large models.

What it solves

LvLLM addresses the memory limitations of GPUs when running large Mixture-of-Experts (MoE) models. It allows these models to span across both VRAM and system memory, preventing out-of-memory errors and enabling the execution of massive models on hardware with limited GPU memory by leveraging CPU resources for computation.

How it works

LvLLM is an extension of vLLM that integrates the lk_moe engine. It implements a hybrid parallel architecture that combines GPU parallelism with NUMA-aware CPU parallelism.

Key mechanisms include:

  • Hybrid Memory Placement: Model weights are distributed between VRAM and pinned NUMA host memory.
  • Flexible Layer Roles: Each MoE layer is assigned a role: GPU-resident (all weights in VRAM), CPU-hybrid (weights in memory, attention in VRAM), or GPU-prefill (large batches on GPU, small on CPU).
  • Computing Modes: It supports three modes—CPU-GPU hybrid decode, hybrid prefill, and pure GPU prefill—to maximize GPU utilization.
  • NUMA Optimization: It uses NUMA thread binding and interleaving to minimize cross-node communication and increase L3 cache hit rates.

Who it’s for

It is designed for developers and researchers deploying large MoE models (such as DeepSeek-V4, Qwen3, or MiniMax) who need to optimize inference performance on systems with limited VRAM but powerful multi-socket CPU/NUMA configurations.

Highlights

  • VRAM + System Memory Scaling: Enables a "1+1=2" memory footprint, allowing models to utilize 100% of available VRAM and system RAM.
  • NUMA-Aware Scheduling: Reduces cross-node communication to as low as 3%.
  • Broad Quantization Support: Compatible with FP8, NVFP4, MXFP4, and AWQ 4-bit symmetric quantization.
  • vLLM Compatibility: Maintains a minimal diff against upstream vLLM, ensuring it remains compatible with stock behavior when hybrid features are disabled.

Related

  • Project
    ztxz16/fastllm

    A high-performance C++ LLM inference engine that enables hybrid deployment of dense and MoE models across GPU, CPU, NUMA, and disk to maximize hardware utilization.

  • Project
    kvcache-ai/ktransformers

    A framework for efficient LLM inference and fine-tuning that uses CPU-GPU heterogeneous computing to run ultra-large MoE models on consumer hardware.

  • Project
    lyogavin/airllm

    AirLLM enables the execution of massive LLMs on low-VRAM consumer GPUs by streaming model layers one at a time instead of loading the entire model into memory.

  • Project
    vllm-project/vllm-ascend

    A hardware plugin that allows the vLLM inference engine to run on Huawei Ascend NPUs, supporting a variety of open-source LLMs and multimodal models.

  • Project
    lucienhuangfu/eLLM

    An LLM inference framework for CPU servers that optimizes long-horizon inference by trading storage for computation to outperform GPUs in long-context scenarios.