vllm-project/guidellm

GuideLLM – SLO‑aware benchmarking for LLM inference

What it is – GuideLLM is a Python‑based platform that lets you evaluate how large language models behave under realistic production workloads. It talks to any OpenAI‑compatible endpoint (including vLLM servers), drives configurable traffic patterns, and records detailed latency and token‑level metrics so you can see whether your deployment meets service‑level objectives (SLOs).

Why you’d use it

• SLO‑driven metrics – captures first‑token latency, inter‑token latency, end‑to‑end latency distributions, and token‑throughput.
• Realistic traffic generation – supports synchronous, concurrent, constant‑rate, Poisson, and sweep profiles, with warm‑up/cool‑down handling.
• Flexible data sources – synthetic text, synthetic images/video, HuggingFace datasets, local JSON/CSV/text files, or custom loaders.
• Multimodal support – can synthesize vision inputs to benchmark VLM serving.
• Standardised outputs – JSON, CSV, HTML, and plot files for dashboards or regression tracking.
• Extensible CLI/API – easy to script or embed in CI pipelines; also provides a container image for Kubernetes/Openshift.

Quick start

# install (Python 3.10‑3.13)
pip install guidellm[recommended]

# start a vLLM server (example model)
vllm serve "neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16"

# run a sweep benchmark (30 s max)
guidellm run \
  --backend kind=openai_http,target=http://localhost:8000 \
  --profile kind=sweep \
  --constraint kind=max_duration,seconds=30 \
  --data kind=synthetic_text,prompt_tokens=256,output_tokens=128

The command prints progress, then writes benchmarks.json/benchmarks.csv (or HTML/plots if requested) to the results directory.

Typical workflows

Goal	Example command snippet
Rate‑limited load test	--profile kind=constant,rate=10 --constraint kind=max_duration,seconds=20
Dataset‑driven evaluation	--data kind=huggingface,source=abisee/cnn_dailymail,load_kwargs.name=3.0.0 --data‑column‑mapper kind=generative_column_mapper,column_mappings.text_column=article
Chat‑completion benchmark	--backend kind=openai_http,target=http://localhost:8000,request_format=/v1/chat/completions --data kind=json_file,path=data.json
Multimodal VLM test	--data kind=synthetic_image,... --data kind=synthetic_text,...
Scenario reuse	guidellm run --config chat --backend kind=openai_http,target=http://localhost:8000

Installation options

• PyPI – pip install guidellm[recommended]
• From source – pip install git+https://github.com/vllm-project/guidellm.git
• Container – ghcr.io/vllm-project/guidellm:<tag> (multi‑arch, supports Docker/Podman/K8s)

Extending the project The repo includes DEVELOPING.md, CONTRIBUTING.md, and a code‑of‑conduct. Developers can add new backends, request formats, or data loaders by following the documented plugin architecture.

License & citation

• Apache 2.0
• Cite with the provided BibTeX entry if you use GuideLLM in research.

---

GuideLLM is a genuine, production‑oriented benchmarking suite for LLM inference, aimed at engineers and ML teams that need SLO‑aware performance data.