guqiong96/Lvllm

LvLLM is a special NUMA extension of vllm that makes full use of CPU and memory resources, reduces GPU memory requirements, and features an efficient GPU parallel and NUMA parallel architecture, supporting hybrid inference for MOE large models.

What it solves

LvLLM addresses the hardware limitations of running large Mixture-of-Experts (MoE) models by allowing them to span across both GPU VRAM and system RAM. It enables the execution of massive models on hardware that lacks sufficient VRAM to hold the entire model, while providing specific optimizations and fixes for newer GPU architectures (SM80, SM86, SM89, and SM120).

How it works

LvLLM integrates the lk_moe engine into vLLM. This hybrid engine is NUMA-aware and schedules MoE experts across the CPU and GPU. It can overlap GPU prefill with hybrid decoding and allows users to specify which expert layers should remain resident in VRAM to maximize performance. It can also be configured to behave exactly like stock vLLM if the hybrid MoE features are disabled.

Who it’s for

It is designed for developers and researchers running large MoE models (such as DeepSeek-V4, Qwen3.8, and GLM-5.3) on heterogeneous hardware or systems where system memory is used to supplement VRAM.

Highlights

  • Hybrid Memory Execution: Spans MoE models across VRAM and system memory to reach nearly 100% VRAM utilization.
  • Broad Hardware Support: Adds support and fixes for NVIDIA SM80, SM86, SM89, and SM120 architectures.
  • NUMA Awareness: Optimizes CPU+GPU scheduling to improve efficiency on multi-socket systems.
  • GPU Prefill: Supports overlapping GPU prefill with hybrid decoding for improved throughput.

Related

  • Project
    ztxz16/fastllm

    A high-performance LLM inference engine designed for local and server deployment, supporting mixed-device execution (GPU/CPU/Disk) and advanced optimization techniques like speculative decoding.

  • Project
    lucienhuangfu/eLLM

    An LLM inference framework for CPU servers that optimizes memory usage to outperform GPUs in long-horizon, multi-turn inference and ultra-long context tasks.

  • Dispatch
    vLLM Multi-LoRA Serving for MoE Models

    vLLM version 0.15.0 introduces optimized Multi-LoRA serving for Mixture of Experts (MoE) models, enabling multiple fine-tuned adapters to share a single GPU to reduce idle compute capacity.

  • Project
    MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks

    A deployment framework for running Qwen3.8-Flash-Next models across dual-DGX Spark nodes using vLLM, featuring automated kernel patching, multi-node synchronization, and optimized KV cache management.

  • Dispatch
    NVIDIA Nemotron 3 Super Support in vLLM

    vLLM now supports NVIDIA Nemotron 3 Super, a 120B parameter hybrid MoE model optimized for multi-agent AI with a 1 million token context window and high inference efficiency.