vllm-project/vllm-ascend

Community maintained hardware plugin for vLLM on Huawei Ascend

vLLM Ascend Plugin – Run vLLM on Huawei Ascend NPUs

What it is – A community‑maintained hardware plugin that lets the high‑throughput LLM serving engine vLLM run directly on Huawei’s Ascend neural‑processing units (NPUs). It implements the “hardware‑pluggable” RFC so the Ascend backend is cleanly decoupled from vLLM’s core.

Why it matters – Ascend NPUs are designed for large‑scale inference and training. By plugging them into vLLM you can serve transformer‑style LLMs, Mixture‑of‑Experts, embedding models, and multimodal LLMs with the same low‑latency, high‑throughput characteristics vLLM provides on GPUs, but leveraging Ascend’s specialized hardware.

Key points from the README

  • Supported hardware – Atlas 800I/800I‑A2/A3 inference series, Atlas A2/A3 training series, and experimental Atlas 300I Duo.
  • Software stack – Linux, Python 3.10‑3.12, CANN 9.1.0, PyTorch 2.10.0 + TorchNPU 2.10.0.post4, and the matching vLLM version.
  • Release cadence – Regular stable releases (e.g., v0.23.0) and release‑candidate builds (v0.26.0rc1, v0.30.0rc1). Each release aligns with a specific vLLM version.
  • Branch model – main tracks the vLLM main branch; releases/vX.Y.Z tracks a particular vLLM version for bug‑fixes only; feature branches live under rfc/….
  • Getting started – Quick‑start and installation guides are provided for each version (links in the table). The README lists the recommended version matrix.
  • Community – Slack channel #SIG-Ascend, a user forum, weekly meetings, and a DeepWiki knowledge base for deeper technical details.
  • License – Apache 2.0.

Typical workflow

  1. Prepare the environment – Install CANN 9.1.0, PyTorch 2.10.0, TorchNPU 2.10.0.post4, and the matching vLLM release.
  2. Install the plugin – Follow the version‑specific installation guide (e.g., pip install vllm-ascend==0.23.0).
  3. Run vLLM – Launch vLLM with the Ascend backend flag (as documented) to serve your model on the NPU.
  4. Tune & monitor – Use the DeepWiki and community resources for performance tuning, large‑scale expert parallelism, and troubleshooting.

Who should use it – Researchers or engineers who already have access to Ascend NPUs (or can request them via Huawei HiDevLab) and want to serve large language models with vLLM’s efficient scheduling, or who need a production‑grade inference stack on Ascend hardware.

All information is taken directly from the repository’s README; no additional features are inferred.

Related

  • Project
    baidu/vLLM-Kunlun

    A hardware plugin that enables vLLM to run on Kunlun XPU, providing optimized kernels and support for a wide range of LLMs and multimodal models.

  • Project
    guqiong96/Lvllm

    An integration of vLLM and lk_moe that enables large MoE models to run across CPU and GPU memory using NUMA-aware scheduling.

  • Project
    UbiquitousLearning/mllm

    A fast, lightweight multimodal LLM inference engine for mobile and edge devices that enables local execution of VLMs and LLMs across various NPUs and CPUs.

  • Project
    ztxz16/fastllm

    A high-performance LLM inference engine designed for local and server deployment, supporting mixed-device execution (GPU/CPU/Disk) and advanced optimization techniques like speculative decoding.

  • Dispatch
    vLLM Adds NVIDIA Vera Rubin NVL72 Support with 7.8× Throughput Gains

    vLLM now runs on NVIDIA Vera Rubin NVL72 GPUs, delivering up to 7.8× higher inference throughput than GB200 NVL72 and showing strong early performance on AgentX and MLPerf benchmarks.