llm-d/llm-d

Achieve state of the art inference performance with modern accelerators on Kubernetes

llm‑d – High‑Performance Distributed LLM Inference on Kubernetes

What it is

  • An open‑source stack that sits on top of model servers such as vLLM and SGLang and adds production‑grade orchestration, routing, and cache management for large‑language‑model (LLM) inference.
  • Built for cloud‑native environments (Kubernetes, Helm) and backed by a CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA.

Core capabilities

Area What llm‑d adds
Intelligent routing Prefix‑cache‑aware and load‑aware load‑balancing; experimental latency‑prediction scheduler to cut latency and boost throughput.
KV‑cache management Tiered off‑loading of the attention KV cache to CPU or disk, with a global index that lets multi‑turn requests keep a larger working set.
Large‑model serving Prefill/decode disaggregation and wide expert‑parallelism that spreads massive models (e.g., DeepSeek‑R1, GPT‑OSS‑120B) across fast inter‑connects.
Operational excellence Flow‑control for multi‑tenant traffic, SLO‑aware autoscaling, and active‑active HA to keep services stable at scale.
Batch processing OpenAI‑compatible batch APIs and async pipelines for offline workloads, maximizing hardware utilisation.

Performance highlights (as reported in the README)

  • 3× higher output throughput & 2× faster time‑to‑first‑token (TTFT) on Llama 3.1 70B with prefix‑cache routing (4× AMD MI300X).
  • 40 % TTFT/ITL reduction using predicted‑latency scheduling on NVIDIA GPUs.
  • Up to 70 % more tokens / sec with prefill/decode disaggregation (GPT‑OSS on AWS B200).
  • 50 k tokens / sec cluster throughput with wide expert‑parallelism on a 16×16 NVIDIA B200 mesh.\nHow you would use it
  1. Deploy – Follow the Quick‑Start guide to install the Helm charts on a Kubernetes cluster (any cloud or on‑prem). The stack provisions the router, cache manager, and model‑server pods automatically.
  2. Configure – Choose a “well‑lit path” (e.g., Optimized Baseline, Tiered Prefix Cache, Wide Expert‑Parallelism) that matches your model size and hardware accelerator (NVIDIA, AMD, Intel, Google TPU, etc.).
  3. Run – Send inference requests via the provided OpenAI‑compatible REST endpoint or the batch API. llm‑d will route traffic, manage KV‑cache tiers, and scale pods based on real‑time latency signals.
  4. Monitor – Built‑in Prometheus metrics and SLO‑aware autoscaling let you keep an eye on throughput, latency, and resource utilisation.

Who benefits

  • Enterprises that need to serve many concurrent LLM users with low latency and predictable cost.
  • ML‑ops teams looking for a reproducible, cloud‑native way to run state‑of‑the‑art models on heterogeneous accelerators.
  • Research labs that want to benchmark new models or inference tricks without hand‑tuning Kubernetes deployments.

Getting started

  • Read the Quickstart Guide: https://llm-d.ai/docs/getting-started/quickstart
  • Pick a starter recipe from the Well‑Lit Paths documentation (e.g., optimized-baseline).
  • Deploy with Helm (the repo provides version‑pinned charts). The README points to the releases page for exact chart versions.

Community & contribution

  • CNCF‑sanctioned project with a public Slack, bi‑weekly stand‑ups, SIGs, and a full contribution guide.
  • Apache 2.0 license – free for commercial and academic use.

All details above are taken directly from the repository’s README; no additional features have been inferred.

Related

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Project
    xLLM-AI/xllm

    xLLM is an open‑source C++ inference framework that delivers high‑throughput, low‑latency LLM serving on Chinese AI accelerators (Ascend, Cambricon, Moore Threads, etc.). It separates a brpc‑based service layer from a hardware‑optimized engine, includes hybrid KV‑cache management via Mooncake, and provides ready‑to‑run deployment scripts for models such as GLM‑5.3‑Flash and MiniMax‑M3. The project is Apache‑2.0 licensed and backed by JD.com and several Chinese university labs.

  • Project
    ai-dynamo/dynamo

    An open-source, datacenter-scale orchestration layer that coordinates multiple inference engines across GPU clusters to maximize throughput and minimize latency for LLMs and multimodal models.

  • Project
    NVIDIA/TensorRT-LLM

    An open-source optimization library from NVIDIA that maximizes inference performance, throughput, and efficiency for LLMs and visual generation models on NVIDIA GPUs.

  • Project
    ztxz16/fastllm

    A high-performance C++ LLM inference engine that enables hybrid deployment of dense and MoE models across GPU, CPU, NUMA, and disk to maximize hardware utilization.