jd-opensource/xllm

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

What it solves

xLLM addresses the challenge of deploying Large Language Models (LLMs) on Chinese AI accelerators, providing an efficient, enterprise-grade inference framework that reduces operational costs while maintaining high performance.

How it works

The framework utilizes a service-engine decoupled architecture, where a dedicated service layer manages scheduling and availability while the engine layer handles the actual computation. It incorporates hybrid KV cache management (based on Mooncake) for intelligent offloading and prefetching to optimize memory and throughput.

Who it’s for

It is designed for enterprises and developers who need to deploy LLMs on specific hardware accelerators such as Ascend NPU, Cambricon MLU, Moore Threads GPU, Hygon DCU, MetaX MACA, and Iluvatar CoreX GPU.

Highlights

  • Hardware Optimization: Deeply optimized for a wide range of Chinese AI accelerators.
  • Decoupled Architecture: Separates service management from the computation engine for better scalability.
  • High Performance: Engineered for high throughput and low latency.
  • Battle-tested: Proven at scale within JD.com's core retail business operations.