ai-dynamo/dynamo

A Datacenter Scale Distributed Inference Serving Framework

What it solves

Dynamo is a datacenter-scale orchestration layer designed to coordinate multiple GPUs and nodes into a single, high-performance inference system. It addresses the limitations of single-node inference engines by providing intelligent routing, disaggregated serving, and automated scaling to maximize throughput and minimize latency for LLM, multimodal, and video generation workloads.

How it works

Dynamo sits above existing inference engines (such as SGLang, TensorRT-LLM, and vLLM) rather than replacing them. It uses a combination of Rust for performance and Python for extensibility to implement several core mechanisms:

  • Disaggregated Serving: Separates the prefill and decode phases of inference into independently scalable GPU pools.
  • KV-Aware Routing: Routes requests based on worker load and KV cache overlap to eliminate redundant prefill computations.
  • KV Block Manager (KVBM): Offloads the KV cache across GPU, CPU, SSD, and remote storage to extend context length.
  • ModelExpress: Streams model weights GPU-to-GPU via NIXL/NVLink for faster cold-starts.
  • SLA-Based Planner: An autoscaler that profiles workloads to right-size GPU pools and meet latency targets.
  • Grove: A Kubernetes operator for topology-aware gang scheduling on NVL72 systems.

Who it’s for

It is intended for developers and operators serving LLMs across multiple GPUs or nodes who need to meet strict latency SLAs, reduce total cost of ownership (TCO), and scale prefill and decode independently.

Highlights

  • High Performance: Demonstrates up to 750x higher throughput for DeepSeek-R1 on specific hardware.
  • Engine Agnostic: Supports SGLang, TensorRT-LLM, and vLLM backends.
  • Zero-Config Deploy: Beta feature allowing deployment via a single YAML specifying model, hardware, and SLA.
  • OpenAI Compatible: Exposes an OpenAI-compatible API for easy integration.
  • Fault Tolerance: Includes canary health checks and in-flight request migration to ensure reliability.

Related

  • Project
    llm-d/llm-d

    A high-performance distributed inference serving stack for Kubernetes that optimizes LLM production deployments across various hardware accelerators.

  • Project
    kaito-project/kaito

    A Kubernetes operator suite that automates the deployment, scaling, and management of LLM inference, fine-tuning, and RAG engines.

  • Project
    NVIDIA/TensorRT-LLM

    An open-source library from NVIDIA that optimizes inference for LLMs and visual generation models using specialized kernels and an efficient runtime to maximize throughput and latency on NVIDIA GPUs.

  • Project
    volcano-sh/kthena

    A Kubernetes-native LLM inference platform that provides declarative model lifecycle management and intelligent routing to make AI infrastructure scalable and cost-efficient.

  • Project
    flashinfer-ai/flashinfer

    A high-performance GPU kernel library and generator for LLM inference that optimizes attention, GEMM, and MoE operations across diverse NVIDIA GPU architectures.