baidu/vLLM-Kunlun

vLLM Kunlun (vllm-kunlun) is a community-maintained hardware plugin designed to seamlessly run vLLM on the Kunlun XPU.

What it solves

It enables the vLLM inference engine to run on Kunlun XPU hardware. By providing a hardware-pluggable interface, it allows users to deploy popular open-source LLMs and multimodal models on Kunlun XPUs without needing to modify the core vLLM source code.

How it works

The project acts as a community-maintained hardware plugin for vLLM. It implements the necessary platform-specific logic, custom operators (such as FlashMLA and fused MoE kernels), and quantization support to bridge the gap between vLLM's high-level engine and the Kunlun XPU architecture. It also includes a "Piecewise Kunlun Graph" for hardware-accelerated graph optimization.

Who it’s for

Developers and AI engineers who use Kunlun3 P800 hardware and want to leverage vLLM's high-performance inference capabilities for a wide variety of generative and multimodal models.

Highlights

  • Broad Model Support: Compatible with 20+ mainstream models including Qwen, Llama, DeepSeek, GLM, and Gemma4.
  • Hardware Acceleration: Features Piecewise Kunlun Graph and optimized attention mechanisms like FlashMLA.
  • Quantization: Supports W8A8, AWQ, GPTQ, and compressed-tensors W4A16.
  • Advanced Decoding: Implements speculative decoding via MTP (Multi-Token Prediction) and DFlash/EAGLE-style proposers.
  • OpenAI Compatible: Provides a standard OpenAI API interface for serving models.

Related

  • Project
    vllm-project/vllm-ascend

    vLLM Ascend is a hardware plugin that enables the vLLM LLM serving engine to run on Huawei Ascend NPUs. It supports a range of Atlas NPU hardware, requires Linux, Python 3.10‑3.12, CANN 9.1.0, PyTorch 2.10.0 + TorchNPU, and matches specific vLLM versions. The project provides regular releases, detailed docs, and an active community (Slack, forum, weekly meetings). It’s licensed under Apache 2.0.

  • Project
    xLLM-AI/xllm

    An efficient LLM inference framework optimized for Chinese AI accelerators to enable high-throughput, low-latency enterprise deployment.

  • Project
    vllm-project/vllm-metal

    vLLM Metal is a plugin that enables the vLLM LLM serving system to run on Apple Silicon Macs using the MLX/Metal backend. It provides Metal‑optimized attention kernels (paged varlen, M5 NAX prefill) and supports large models such as Qwen‑3.8‑27B. Install via Homebrew or a curl script, then use the regular `vllm` command on macOS 15+.

  • Project
    vllm-project/tpu-inference

    vLLM TPU is an open‑source plugin that lets you run PyTorch or JAX LLMs on Google TPUs using the same vLLM interface. It supports high‑performance features (async scheduling, KV‑cache offload, LoRA, speculative decoding, etc.) and is tested on many top models such as Gemma‑4, Llama‑3.1, and Qwen 3. Recommended TPUs are v7x, v5e, and v6e. The project provides quick‑start docs, a support matrix, and an active community for contributors.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.