sgl-project/genai-bench

Genai-bench is a powerful benchmark tool designed for comprehensive token-level performance evaluation of large language model (LLM) serving systems.

What it solves

Genai-bench provides a unified way to evaluate the token-level performance of LLM serving systems. It eliminates the need for fragmented tools by offering a standardized method to measure throughput, latency, and error rates across different traffic scenarios and concurrency levels.

How it works

The tool operates through a CLI that allows users to initiate benchmarks against model backends via API. It supports both text-to-text and text-to-image tasks. Once the benchmark is run, it collects raw metrics and logs, which can then be processed into detailed Excel reports and visual plots for performance analysis.

Who it’s for

It is designed for developers and engineers working on LLM serving infrastructure who need to measure and optimize the performance of their model deployment systems.

Highlights

  • Live UI Dashboard: Real-time monitoring of progress, logs, and metrics during a benchmark run.
  • Comprehensive Metrics: Tracks Time to First Token (TTFT), End-to-End (E2E) latency, Time Per Output Token (TPOT), throughput, and requests per second (RPS).
  • Experiment Analyzer: Generates Excel reports and customizable plot grids for comparing different experiments.
  • Multi-modal Support: Capable of benchmarking both text generation and image generation APIs.

Related

  • Project
    ai-dynamo/aiperf

    A comprehensive benchmarking tool for generative AI models that measures latency and throughput across various inference solutions and modalities.

  • Project
    AISBench/benchmark

    AISBench Benchmark 是一个用于 AI 模型的性能与精度评测工具，支持文本、多模态和智能体场景，可对接 vLLM 等服务化推理后端，内置大量数据集与评测基准。

  • Project
    MatthewCYM/VoiceBench

    VoiceBench is a benchmark suite for evaluating LLM‑based voice assistants. It supplies a Hugging‑Face dataset with audio‑driven QA and instruction‑following tasks, a command‑line toolkit to run a model, collect spoken responses, and automatically score them (via GPT‑4o‑mini for open‑ended sets). Results are posted on a public leaderboard, and the repo includes a curated list of recent voice‑assistant papers.

  • Project
    aisa-group/PostTrainBench

    PostTrainBench is a benchmark that measures how well autonomous LLM agents can fine‑tune a base language model within a 10‑hour H100 budget, using four CLI scaffolds (Claude, Codex, Gemini, OpenCode) across six diverse downstream tasks. It provides an HTCondor reference pipeline and a cloud‑GPU adapter, with detailed prompts, environment setup, and support for both API‑key and subscription‑based agents.

  • Project
    vllm-project/guidellm

    GuideLLM is a Python library (also available as a container) that benchmarks LLM inference against real‑world workloads. It drives configurable traffic patterns to OpenAI‑compatible or vLLM servers, supports synthetic and real multimodal datasets, records detailed latency/token metrics, and outputs JSON/CSV/HTML reports for capacity planning and SLO verification.