llm-d/llm-d

State of the art inference performance with modern accelerators on Kubernetes

What it solves

llm-d is designed to eliminate the complexity and manual tuning required to achieve state-of-the-art (SOTA) inference performance for large language models (LLMs) in production. It provides a high-performance distributed inference serving stack optimized for Kubernetes, ensuring that OSS models can be served efficiently and reliability across various hardware accelerators.

How it works

llm-d acts as an orchestration and optimization layer that sits above model servers like vLLM and SGLang. It employs several advanced techniques to maximize throughput and reduce latency:

  • Intelligent Routing: Uses prefix-cache and load-aware balancing, along with experimental predicted-latency scheduling to optimize request distribution.
  • KV-Cache Management: Implements tiered offloading to CPU or disk and global indexing to increase the effective working set size for multi-turn conversations.
  • Serving Large Models: Utilizes prefill/decode disaggregation and wide expert-parallelism to optimize massive models like DeepSeek-R1.
  • Operational Excellence: Provides SLO-aware autoscaling and intelligent flow control for multi-tenant environments.
  • Batch Processing: Manages large-scale offline inference via OpenAI-compatible Batch APIs.

Who it’s for

MLOps engineers and infrastructure providers who need to deploy and scale LLM inference services on Kubernetes across diverse hardware (NVIDIA, AMD, Intel, Google TPU) and cloud providers.

Highlights

  • CNCF Sandbox Project: Founded by industry leaders including Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.
  • Hardware Agnostic: Optimized for a wide range of accelerators including NVIDIA B200, AMD MI300X, and Intel XPU.
  • Prefill/Decode Disaggregation: Significant throughput gains (up to 70% higher tokens/sec) compared to standard vLLM.
  • Hierarchical KV Offloading: Up to 13.9x throughput improvement for high concurrent user loads.
  • Detailed Benchmarks: Provides reproducible benchmarks via the Prism platform.

Related

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Project
    ztxz16/fastllm

    A high-performance LLM inference engine designed for local and server deployment, supporting mixed-device execution (GPU/CPU/Disk) and advanced optimization techniques like speculative decoding.

  • Project
    kkokosa/dotLLM

    A high-performance LLM inference engine written natively in C#/.NET, providing native CPU/GPU support for transformer models without relying on Python or llama.cpp.

  • Project
    xLLM-AI/xllm

    An efficient LLM inference framework optimized for Chinese AI accelerators to enable high-throughput, low-latency enterprise deployment.

  • Project
    Tencent/hpc-ops

    A production-grade operator library for LLM inference that provides highly optimized CUDA kernels for attention, MoE, and sampling to reduce serving latency.