vllm-project/guidellm

What it solves

GuideLLM addresses the difficulty of evaluating how Large Language Models (LLMs) perform under real-world production workloads. Many existing tools benchmark general API endpoints rather than the specific nuances of LLM inference, often missing critical token-level metrics and the ability to simulate complex, realistic traffic patterns required for Service Level Objective (SLO) planning and capacity planning.

How it works

GuideLLM simulates end-to-end interactions with OpenAI-compatible or vLLM-native servers. It generates specific workload patterns (such as constant, Poisson, or concurrent streams) and feeds them into the server using either real datasets (from HuggingFace or local files) or synthetic data (including text, images, and video). The platform then captures detailed latency and token-level statistics—such as Time to First Token (TTFT) and Inter-Token Latency (ITL)—and produces standardized reports in JSON, CSV, and HTML formats.

Who it’s for

It is designed for engineering and ML teams who need to tune model deployments, assess system behavior under load, and plan hardware capacity for LLM inference in production environments.

Highlights

• SLO-aware metrics: Captures full distributions for TTFT, ITL, and end-to-end latency.
• Flexible traffic simulation: Supports various load profiles including synchronous, concurrent, throughput-based, and automated sweeps to find operating limits.
• Multimodal support: Capable of benchmarking text, image, audio, and video inputs using both real and synthetic datasets.
• Extensible execution: High-throughput benchmarking powered by multiprocessing, threading, and async execution.
• Standardized reporting: Generates exportable reports for dashboards and regression tracking.