vllm-project/vllm-ascend
Community maintained hardware plugin for vLLM on Ascend
What it solves
It enables the vLLM inference engine to run seamlessly on Huawei Ascend NPUs (Neural Processing Units). By providing a hardware-pluggable interface, it decouples the Ascend NPU integration from the core vLLM codebase, allowing users to deploy high-performance LLMs on Ascend hardware without modifying the main engine.
How it works
The project acts as a community-maintained hardware plugin. It implements a pluggable interface (based on a specific RFC) that allows vLLM to communicate with Ascend hardware. This enables support for various model architectures, including Transformer-like models, Mixture-of-Experts (MoE), Embedding models, and Multi-modal LLMs, using the CANN software stack and PyTorch.
Who it’s for
Developers and researchers who have access to Ascend NPU hardware (such as Atlas 800I or Atlas A2/A3 series) and want to use vLLM for efficient model inference and deployment.
Highlights
- Broad Model Support: Compatible with Transformer, MoE, Embedding, and Multi-modal LLMs.
- Hardware Pluggability: Decouples hardware integration from the core vLLM engine for easier maintenance.
- Enterprise Hardware Support: Specifically optimized for Atlas 800I and Atlas A2/A3 series NPUs.
- Community Driven: Maintained as a plugin to ensure compatibility with vLLM's main release cycle.