vllm-project/vllm-ascend

Community maintained hardware plugin for vLLM on Ascend

What it solves

It enables the vLLM inference engine to run seamlessly on Huawei Ascend NPUs, decoupling the hardware integration from the core vLLM codebase through a pluggable interface.

How it works

It acts as a hardware plugin that implements a specific interface (based on a hardware-pluggable RFC) to bridge vLLM's requirements with the Ascend NPU architecture. This allows various model types—including Transformer-based LLMs, Mixture-of-Experts (MoE), Embedding models, and Multi-modal LLMs—to be executed on Ascend hardware.

Who it’s for

Developers and AI engineers who use Huawei Ascend hardware (such as Atlas 800I or Atlas A2/A3 series) and want to deploy high-performance LLM inference using the vLLM framework.

Highlights

  • Support for a wide range of model architectures including MoE and multimodal LLMs.
  • Adheres to a hardware-pluggable architecture to keep the main vLLM project clean.
  • Compatible with CANN 9.1.0 and specific PyTorch/TorchNPU versions.
  • Community-maintained with dedicated support for various Atlas NPU series.

Related

  • Project
    baidu/vLLM-Kunlun

    A hardware plugin that enables vLLM to run on Kunlun XPU, providing optimized kernels and support for a wide range of LLMs and multimodal models.

  • Project
    guqiong96/Lvllm

    An extension of vLLM that enables hybrid GPU and NUMA-aware CPU inference for MoE models, allowing them to span across VRAM and system memory.

  • Project
    ztxz16/fastllm

    A high-performance C++ LLM inference engine that enables hybrid deployment of dense and MoE models across GPU, CPU, NUMA, and disk to maximize hardware utilization.

  • Project
    xLLM-AI/xllm

    xLLM is an open‑source C++ inference framework that delivers high‑throughput, low‑latency LLM serving on Chinese AI accelerators (Ascend, Cambricon, Moore Threads, etc.). It separates a brpc‑based service layer from a hardware‑optimized engine, includes hybrid KV‑cache management via Mooncake, and provides ready‑to‑run deployment scripts for models such as GLM‑5.3‑Flash and MiniMax‑M3. The project is Apache‑2.0 licensed and backed by JD.com and several Chinese university labs.

  • Project
    UbiquitousLearning/mllm

    A fast, lightweight multimodal LLM inference engine for mobile and edge devices that optimizes vision-language model execution across various NPUs and GPUs.