sgl-project/genai-bench

Genai-bench is a powerful benchmark tool designed for comprehensive token-level performance evaluation of large language model (LLM) serving systems.

What it solves

Genai-bench provides a unified way to evaluate the token-level performance of LLM serving systems. It eliminates the need for fragmented tools by offering a standardized method to measure throughput, latency, and error rates across different traffic scenarios and concurrency levels.

How it works

The tool operates through a CLI that allows users to initiate benchmarks against model backends via API. It supports both text-to-text and text-to-image tasks. Once the benchmark is run, it collects raw metrics and logs, which can then be processed into detailed Excel reports and visual plots for performance analysis.

Who it’s for

It is designed for developers and engineers working on LLM serving infrastructure who need to measure and optimize the performance of their model deployment systems.

Highlights

  • Live UI Dashboard: Real-time monitoring of progress, logs, and metrics during a benchmark run.
  • Comprehensive Metrics: Tracks Time to First Token (TTFT), End-to-End (E2E) latency, Time Per Output Token (TPOT), throughput, and requests per second (RPS).
  • Experiment Analyzer: Generates Excel reports and customizable plot grids for comparing different experiments.
  • Multi-modal Support: Capable of benchmarking both text generation and image generation APIs.

Related

  • Project
    ai-dynamo/aiperf

    A comprehensive benchmarking tool for generative AI models that measures latency and throughput across various inference solutions and load patterns.

  • Project
    AISBench/benchmark

    A model evaluation tool based on OpenCompass that provides accuracy and performance benchmarking for both local and service-based AI models across text and multimodal tasks.

  • Project
    MatthewCYM/VoiceBench

    VoiceBench is a benchmarking framework and dataset for evaluating LLM-based voice assistants across tasks like reasoning, safety, and open-ended QA.

  • Project
    aisa-group/PostTrainBench

    A benchmark that measures the ability of CLI agents to autonomously post-train pre-trained LLMs to improve their performance on specific benchmarks.

  • Project
    vllm-project/guidellm

    An SLO-aware benchmarking and evaluation platform that simulates real-world workloads to optimize LLM inference performance, resource needs, and operational limits.