casys-kaist/LLMServingSim

LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure

What it solves

LLMServingSim addresses the difficulty of testing and optimizing LLM serving infrastructure without needing massive, expensive physical hardware clusters. It allows researchers to simulate how different hardware configurations, memory tiers, and parallelism strategies affect the performance of LLM inference at scale.

How it works

The project uses a cycle-level simulation approach combining three main components:

  1. Python Frontend: Mimics the continuous-batching scheduler used in vLLM.
  2. C++ Backend: Uses ASTRA-Sim for analytical network simulation.
  3. Layerwise Profiler: Feeds the system with real per-hardware latency data captured from vLLM to ensure simulation accuracy.

Who it’s for

It is designed for researchers and engineers studying LLM infrastructure, specifically those focusing on heterogeneous accelerators, disaggregated memory (such as CXL or PIM), and large-scale parallelism techniques.

Highlights

  • Broad Parallelism Support: Simulates Tensor Parallelism (TP), Pipeline Parallelism (PP), Expert Parallelism (EP), and Data Parallelism (DP) in various combinations.
  • Hardware Flexibility: Supports heterogeneous accelerators and disaggregated memory tiers (CPU, CXL, PIM).
  • High Accuracy: Profiles for hardware like the RTX 4090 show results within 1% of real vLLM runs for latency and TTFT/TPOT.
  • Efficient Performance: Recent updates have increased simulation speed by approximately 11x while maintaining byte-identical results.

Related

  • Project
    vllm-project/semantic-router

    A programmable routing layer for Mixture-of-Models systems that optimizes LLM request routing across heterogeneous compute and infrastructure to improve cost, latency, and quality.

  • Project
    ai-dynamo/aiconfigurator

    A configuration optimization tool that helps users find the best GPU and parallelism settings for disaggregated LLM serving to maximize throughput while meeting latency SLAs.

  • Project
    sgl-project/sglang

    A high-performance serving framework for LLMs and multimodal models that provides low-latency, high-throughput inference across single GPUs and large distributed clusters.

  • Project
    llm-d/llm-d

    llm‑d is an open‑source, CNCF‑sandbox project that adds production‑grade routing, KV‑cache tiering, and expert‑parallelism to existing LLM model servers (vLLM, SGLang). Delivered as Helm charts for Kubernetes, it lets organizations serve large language models at scale with lower latency and higher throughput across many accelerator types.

  • Project
    b4rtaz/distributed-llama

    A distributed inference engine that clusters home devices over Ethernet to share RAM and accelerate LLM performance using tensor parallelism.