ai-dynamo/aiperf

AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution.

What it solves

AIPerf is a benchmarking tool designed to measure the performance of generative AI models across various inference solutions. It helps developers and researchers determine the exact performance characteristics of their model deployments, such as latency, throughput, and how the system handles different load patterns.

How it works

AIPerf uses a scalable multiprocess architecture with 10 services communicating via ZMQ to simulate user traffic. It can be configured via CLI flags or YAML files to send requests to supported APIs (including OpenAI-compatible endpoints, NIM, and Hugging Face TGI). It supports multiple benchmarking modes, including fixed concurrency, request-rate control, and the replay of real-world production traces (e.g., from SageMaker or Baseten).

Who it’s for

It is intended for AI engineers, ML Ops professionals, and researchers who need to rigorously test the performance and scalability of LLM, image, audio, and video generation endpoints before or during production deployment.

Highlights

  • Comprehensive Metrics: Tracks Time to First Token (TTFT), Inter Token Latency (ITL), and overall request throughput.
  • Linguistic & Multimodal Support: Benchmarks text LLMs, embeddings, rankings, audio, image generation, and video generation.
  • Advanced Load Control: Supports Poisson and gamma traffic arrival patterns, gradual ramping, and warmup phases to eliminate cold-start effects.
  • Production Trace Replay: Ability to replay deterministic workloads from public datasets (ShareGPT) or production logs.
  • Telemetry Integration: Integrates with MLflow, OpenTelemetry, and Weights & Biases for live metric streaming and result uploads.
  • Extensible Architecture: Features a plugin system for custom endpoints, datasets, and metrics.

Related

  • Project
    mlcommons/inference

    A standardized benchmark suite for measuring the speed and efficiency of AI model inference across various hardware and deployment scenarios.

  • Project
    flagos-ai/FlagPerf

    An integrated AI hardware evaluation engine that benchmarks the actual performance of AI chips across various models, frameworks, and software stacks to ensure fair and reproducible results.

  • Project
    mlcommons/training

    A repository of reference (unoptimized) implementations for the MLPerf Training benchmark suite, covering vision, language, recommendation, and graph models. Each benchmark includes model code, Dockerfiles, dataset scripts, and a training‑run script that measures time to reach target quality. Intended for benchmark submissions, hardware evaluation, and learning, but not for real‑world performance.

  • Project
    sgl-project/genai-bench

    A performance benchmarking tool for LLM serving systems that provides token-level metrics, real-time monitoring, and detailed visual reports.

  • Project
    AISBench/benchmark

    A model evaluation tool based on OpenCompass that provides accuracy and performance benchmarking for both local and service-based AI models across text and multimodal tasks.