microsoft/vidur

Accurate, large-scale, and extensible simulator for LLM inference Systems

What it solves

Vidur is a high-fidelity simulator for LLM inference systems. It allows developers and researchers to analyze system performance, plan capacity for deployments, and test new scheduling algorithms or optimizations (like speculative decoding) without requiring constant access to expensive GPUs.

How it works

Vidur simulates the execution of LLM workloads across various hardware configurations (such as A100 or H100 GPUs) and model architectures (like Llama-3 or Qwen). It uses a profiling phase to gather data, then simulates the inference process to predict metrics such as Time to First Token (TTFT), Time Per Output Token (TPOT), and end-to-end request time. It supports complex deployment setups, including Tensor Parallelism (TP) and Pipeline Parallelism (PP).

Who it’s for

It is designed for ML system researchers and engineers who need to optimize LLM serving infrastructure, determine the most cost-effective hardware configurations, or experiment with new inference scheduling strategies.

Highlights

  • GPU-free simulation: Analyze performance and capacity without needing active GPU hardware after an initial profiling step.
  • Hardware and Model Support: Compatible with multiple GPU SKUs (A100, H100, A40) and popular models including Llama-2, Llama-3, CodeLlama, InternLM, and Qwen.
  • Detailed Metrics: Tracks key performance indicators like batch size, TTFT, and TPOT, with outputs exportable to wandb or Chrome traces for visualization.
  • Flexible Configuration: Supports various request generators (synthetic or trace-based) and scheduling algorithms (e.g., Sarathi).

Related

  • Project
    NVIDIA/TensorRT-LLM

    An open-source optimization library from NVIDIA that maximizes inference performance, throughput, and efficiency for LLMs and visual generation models on NVIDIA GPUs.

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Project
    tile-ai/TileRT

    A tile-based runtime engine for ultra-low-latency LLM inference that minimizes time per output token for massive models on NVIDIA B200 GPUs.

  • Project
    casys-kaist/LLMServingSim

    A cycle-level simulator for LLM serving infrastructure that models heterogeneous hardware and parallelism strategies to optimize inference performance.

  • Project
    kkokosa/dotLLM

    A high-performance LLM inference engine written natively in C#/.NET, providing native CPU/GPU support for transformer models without relying on Python or llama.cpp.