baidu/vLLM-Kunlun

What it solves

vLLM Kunlun is a hardware plugin that allows the vLLM inference engine to run on Kunlun XPU hardware. It removes the need to modify the core vLLM source code to support this specific hardware backend, providing a seamless way to deploy popular open-source LLMs on Kunlun processors.

How it works

The project implements a hardware-pluggable interface that decouples the Kunlun XPU integration from the main vLLM codebase. It uses Python entry points to act as a standard platform plugin. To achieve high performance, it includes custom C++ extensions, hardware-accelerated graph optimizations (Piecewise Kunlun Graph), and specialized kernels for attention (FlashMLA) and Mixture-of-Experts (MoE).

Who it’s for

Developers and organizations using Kunlun3 P800 hardware who want to deploy high-performance LLM inference using the vLLM framework and an OpenAI-compatible API.

Highlights

• Broad Model Support: Supports over 20 mainstream models, including the Qwen, Llama, DeepSeek, and Gemma4 series, as well as multimodal models like InternVL.
• Quantization: Supports multiple quantization methods including W8A8 (INT8), AWQ, GPTQ, and compressed-tensors W4A16.
• Advanced Inference Techniques: Implements speculative decoding (MTP and DFlash/EAGLE), Tensor Parallelism for multi-device execution, and Multi-LoRA adapter support.
• ** regulars OpenAI API**: Provides a standard OpenAI-compatible API server for easy integration into existing workflows.