xLLM-AI/xllm

xLLM – High‑Performance LLM Inference for Chinese AI Accelerators

What it is – xLLM is an open‑source inference engine built in C++ that targets large language models (LLMs) running on a variety of Chinese‑origin AI chips (Ascend NPU, Cambricon MLU, Moore Threads MUSA, Hygon DCU, MetaX MACA, Iluvatar CoreX). It separates the service (request scheduling, load‑balancing) from the engine (tensor computation), letting enterprises deploy LLM services at scale with low latency and high throughput.

Key capabilities

• Hardware‑specific optimizations – hand‑tuned kernels and memory management for each supported accelerator, leveraging the latest driver stacks (e.g., Ascend HDK 25.2).
• Hybrid KV‑cache management – integrates the Mooncake library to off‑load and pre‑fetch KV cache entries intelligently, reducing memory pressure for long‑context generation.
• Service‑engine decoupling – a lightweight HTTP/gRPC front‑end built on Apache brpc handles request routing, while the engine focuses purely on computation.
• Model support – day‑0 deployment scripts are provided for a growing list of Chinese‑focused models such as GLM‑5.3‑Flash, MiniMax‑M3, DeepSeek‑V4‑Flash, and the GLM‑4.x series.
• Enterprise‑grade reliability – the framework has been battle‑tested inside JD.com’s retail backend, with tooling for monitoring, rolling upgrades, and multi‑node scaling.

Supported hardware

Chip family	Abbreviation	Example	Notes
Ascend NPU	NPU	A2, A3	Requires HDK driver ≥ 25.2
Cambricon MLU	MLU	MLU
Moore Threads GPU	MUSA	S5000
Hygon DCU	DCU	BW1000
MetaX MACA	MACA	MXC500
Iluvatar CoreX GPU	ILU	BI150

Getting started

• Quick‑start guide – step‑by‑step tutorial to pull a Docker image, launch the service, and run a test request: https://docs.xllm-ai.com/en/getting_started/quick_start/
• Launch scripts – detailed commands for starting the service layer and the engine on a single node or a cluster.
• Offline inference – instructions for batch processing without the service front‑end.
• Supported models list – a catalog of models with ready‑made deployment scripts (e.g., run_glm_53_flash.sh).

Community & support

• Official documentation site: https://docs.xllm-ai.com/.
• Docker images hosted on Quay: quay.io/repository/jd_xllm/xllm-ai.
• Technical report on arXiv (arXiv:2510.14686) explains the architecture and performance results.
• Chinese‑language WeChat group (QR code in the repo) for user questions and announcements.
• The project is now a donation to the OpenAtom Foundation, indicating a move toward broader open‑source governance.

License – Apache License 2.0 (per the header in the repository).

Citation

@article{liu2025xllm,
  title={xLLM Technical Report},
  author={xLLM team},
  journal={arXiv preprint arXiv:2510.14686},
  year={2025}
}

---

All information above is taken directly from the repository’s README; no additional features have been inferred.