xLLM-AI/xllm

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

What it solves

xLLM is designed to reduce the cost and increase the efficiency of deploying large language models (LLMs) on enterprise-grade hardware, specifically targeting Chinese AI accelerators. It provides a high-throughput, low-latency inference framework that is battle-tested at scale within JD.com's retail operations.

How it works

The framework uses a service-engine decoupled architecture. The service layer manages scheduling and availability, while the engine layer handles the actual computation. It also incorporates hybrid KV cache management (based on Mooncake) for intelligent offloading and prefetching to further optimize performance.

Who it’s for

It is intended for enterprises and developers who need to deploy LLMs on Chinese AI hardware such as Ascend NPU, Cambricon MLU, Moore Threads GPU, Hygon DCU, MetaX MACA, and Iluvatar CoreX GPU.

Highlights

  • Broad Hardware Support: Deeply optimized for a wide range of Chinese AI accelerators.
  • Enterprise-Grade: Proven at scale in real-world retail business environments.
  • Decoupled Architecture: Separates service scheduling from the computation engine.
  • High Performance: Focuses on delivering high throughput and low latency for inference.

Related

  • Project
    ztxz16/fastllm

    A high-performance LLM inference engine designed for local and server deployment, supporting mixed-device execution (GPU/CPU/Disk) and advanced optimization techniques like speculative decoding.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.

  • Project
    kkokosa/dotLLM

    A high-performance LLM inference engine written natively in C#/.NET, providing native CPU/GPU support for transformer models without relying on Python or llama.cpp.

  • Project
    localai-org/vllm.cpp

    A high-performance C++20 inference engine that mirrors vLLM's features and throughput but removes the Python/PyTorch dependency for a 140x smaller deployment footprint.

  • Project
    vllm-project/vllm

    vLLM is an open‑source library for high‑throughput, low‑latency serving of large language models. It introduces PagedAttention for efficient KV‑cache handling, supports many quantisation formats, integrates fast attention kernels, and offers an OpenAI‑compatible API that runs on GPUs, CPUs and other accelerators.