casys-kaist/LLMServingSim

LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure

What it solves

LLMServingSim addresses the difficulty of predicting how large language model (LLM) serving systems will perform on complex, heterogeneous hardware. It allows researchers and engineers to simulate how different hardware configurations, memory architectures, and parallelization strategies affect inference latency and throughput without needing to build the physical infrastructure first.

How it works

It functions as a cycle-level simulator that combines three main components: a Python-based frontend that mimics the continuous-batching scheduler found in vLLM, an analytical network backend (ASTRA-Sim) that models communication, and a layerwise profiler that captures latency data from real hardware. By integrating these, the simulator can model complex scenarios like MoE (Mixture of Experts) routing, speculative decoding, and various parallelism techniques (TP, PP, EP, DP) across diverse hardware setups.

Who it’s for

This tool is designed for systems researchers, hardware architects, and infrastructure engineers working on LLM inference optimization, data center resource planning, and the development of high-performance serving stacks.

Highlights

  • Cycle-level simulation of LLM serving infrastructure.
  • Supports heterogeneous hardware and disaggregated memory tiers (CPU, CXL, PIM).
  • Validated against real-world vLLM benchmarks with high accuracy (e.g., within 1% for RTX 4090 profiles).
  • Models advanced serving techniques including MoE routing, speculative decoding, and multi-instance parallelism (TP/PP/EP/DP).

Related

  • Project
    ServerlessLLM/ServerlessLLM

    A high-performance system for deploying multiple AI models on shared GPUs, featuring ultra-fast checkpoint loading and GPU multiplexing to enable efficient serverless inference.

  • Project
    ztxz16/fastllm

    A high-performance LLM inference engine designed for local and server deployment, supporting mixed-device execution (GPU/CPU/Disk) and advanced optimization techniques like speculative decoding.

  • Project
    triton-inference-server/tensorrtllm_backend

    A Triton Inference Server backend that enables high-performance serving of TensorRT-LLM optimized models with support for in-flight batching and multi-GPU scaling.

  • Project
    llm-d/llm-d

    A high-performance distributed inference serving stack for Kubernetes that optimizes LLM production deployments across various hardware accelerators.

  • Dispatch
    vLLM Semantic Router: Enhancing Model Performance via Micro-Agent Collaboration

    vLLM introduces the Semantic Router, a serving-layer primitive that transforms a single model API call into a bounded collaboration of micro-agents to outperform frontier models on complex benchmarks.