ai-dynamo/aiconfigurator

Offline optimization of your disaggregated Dynamo graph

What it solves

Configuring LLM deployments for disaggregated serving is complex because users must balance the number of prefill and decode workers, parallelism settings, and SLA targets (like Time to First Token and Time per Output Token) to maximize throughput without sacrificing latency.

How it works

AIConfigurator models LLM inference using collected performance data for specific hardware and frameworks. It searches through thousands of potential configurations to find the optimal setup based on the user's model, GPU count, and GPU type. It can operate in several modes:

  • Default: Compares aggregated vs. disaggregated deployments to find the best overall configuration.
  • Recommend: Acts as a procurement tool to calculate the minimum GPU count needed to meet specific performance targets.
  • Generate: Quickly creates a basic configuration without a full parameter sweep.
  • Exp: Runs custom experiments defined in YAML files.

Who it’s for

ML engineers and infrastructure architects who need to optimize LLM serving deployments, specifically those using Dynamo or llm-d, to achieve the highest possible throughput while meeting strict latency SLAs.

Highlights

  • Multi-Backend Support: Works with trtllm, vllm, and sglang.
  • Automated Search: Evaluates thousands of configurations to find the Pareto frontier of throughput vs. latency.
  • Deployment Artifacts: Generates configuration files for various targets including Dynamo manifests, llm-d Helm/Kustomize, and FPM.
  • Procurement Sizing: Recommends the minimum hardware required to hit a specific request rate and SLA.

Related

  • Project
    llm-d/llm-d

    llm‑d is an open‑source, CNCF‑sandbox project that adds production‑grade routing, KV‑cache tiering, and expert‑parallelism to existing LLM model servers (vLLM, SGLang). Delivered as Helm charts for Kubernetes, it lets organizations serve large language models at scale with lower latency and higher throughput across many accelerator types.

  • Project
    casys-kaist/LLMServingSim

    A cycle-level simulator for LLM serving infrastructure that models heterogeneous hardware and parallelism strategies to optimize inference performance.

  • Project
    sgl-project/sglang

    A high-performance serving framework for LLMs and multimodal models that provides low-latency, high-throughput inference across single GPUs and large distributed clusters.

  • Dispatch
    Optimizing LLM Performance: Solving Long Prompt Blocking and Decode Slowdowns

    Hugging Face explores how long prompts block request queues and slow down token generation, proposing request-parallel prefills and disaggregated prefill as solutions to reduce latency.

  • Project
    ai-dynamo/dynamo

    An open-source, datacenter-scale orchestration layer that coordinates multiple inference engines across GPU clusters to maximize throughput and minimize latency for LLMs and multimodal models.