ai-dynamo/dynamo

A Datacenter Scale Distributed Inference Serving Framework

Dynamo – a datacenter‑scale inference orchestration stack

What it is – Dynamo is an open‑source platform that sits above existing LLM inference engines (vLLM, SGLang, TensorRT‑LLM, etc.) and turns a collection of GPUs or whole nodes into a coordinated, multi‑node serving system. It does not replace the underlying engines; instead it adds orchestration, routing, caching and autoscaling so that large language‑model workloads can be served at scale with low latency.

Key capabilities

Feature What it does Why it matters
Disaggregated prefill / decode Prefill (prompt processing) and decode (token generation) run on separate GPU pools that can be scaled independently. Keeps GPUs busy – prefill is compute‑heavy, decode is memory‑heavy – so overall throughput improves.
KV‑aware routing Requests are routed based on the overlap of their key‑value (KV) cache with workers’ existing caches. Avoids repeating the same prefill work, cutting time‑to‑first‑token (TTFT) roughly in half.
KV Block Manager (KVBM) Offloads KV cache from GPU to CPU, SSD or remote storage, extending effective context length beyond GPU memory.
ModelExpress Streams model weights directly between GPUs over NVLink/NIXL. Enables cold‑starts up to 7× faster when launching new replicas.
Planner (SLA‑driven autoscaler) Profiles workloads, predicts latency, and automatically right‑sizes the prefill and decode pools to meet latency Service‑Level‑Agreements at minimal cost.
Grove K8s operator Provides topology‑aware gang scheduling for NVIDIA‑NVL72 racks.
AISimulate Offline simulator that predicts serving behavior and searches for optimal deployment configurations before any GPU is allocated.
Fault tolerance Canary health checks and in‑flight request migration keep requests alive when a worker fails.

Supported back‑ends – Dynamo works with SGLang, TensorRT‑LLM and vLLM (all marked ✅ in the feature matrix). It also supports multimodal models, tool‑calling, and video‑generation pipelines.

How you use it

  1. Container quick‑start – Pull a pre‑built Docker image for the desired backend, run the Dynamo frontend and a worker, then call the OpenAI‑compatible /v1/chat/completions endpoint.
  2. Python install – Install the ai-dynamo package (via uv or pip) with the extra for your backend, then launch the same components from the command line.
  3. Kubernetes production – Deploy the Dynamo Platform and submit a single DynamoGraphDeploymentRequest manifest that specifies model, backend and latency SLA. Dynamo’s planner, AISimulate and Grove automatically provision the right number of GPU pods and configure routing.

Performance highlights from the README

  • 7× higher throughput per GPU on DeepSeek‑R1 (GB200 NVL72) vs a baseline without Dynamo.
  • 7× faster model cold‑start using ModelExpress weight streaming.
  • 2× faster TTFT on Qwen3‑Coder 480B thanks to KV‑aware routing.
  • 80 % fewer SLA breaches with planner‑driven autoscaling.
  • Up to 750× throughput increase on DeepSeek‑R1 on a larger GB300 system.

Community & ecosystem – Dynamo is backed by NVIDIA, has a public Slack, regular office‑hours, a design‑proposal process, and more than 160 contributors. It integrates with LangChain, NVIDIA NeMo Agent Toolkit, and provides recipes for popular models (Qwen, DeepSeek, Kimi, etc.).

When to reach for Dynamo – If you need to serve a large model across multiple GPUs or nodes, want to avoid redundant prefill work, need automatic scaling to meet latency SLAs, or want to extend context length beyond GPU memory, Dynamo provides the orchestration layer to make that feasible. For a single‑GPU, single‑model deployment, a plain inference engine is usually sufficient.

TL;DR – Dynamo is an open‑source, Rust‑based orchestration framework that turns existing LLM inference engines into a scalable, low‑latency, multi‑node serving system with smart routing, KV caching, autoscaling and Kubernetes integration.

Related

  • Project
    llm-d/llm-d

    A high-performance distributed inference serving stack for Kubernetes that optimizes LLM production deployments across various hardware accelerators.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    sgl-project/sglang

    An open-source inference framework optimized for fast serving of LLMs, vision-language, and diffusion models, particularly for agentic workloads and RL rollouts.

  • Project
    guqiong96/Lvllm

    An integration of vLLM and lk_moe that enables large MoE models to run across CPU and GPU memory using NUMA-aware scheduling.

  • Project
    triton-inference-server/tensorrtllm_backend

    A Triton Inference Server backend that enables high-performance serving of TensorRT-LLM optimized models with support for in-flight batching and multi-GPU scaling.