ai-dynamo/aiperf

What it solves

AIPerf is a benchmarking tool designed to measure the performance of generative AI models. It helps developers and operators determine how their chosen inference solution handles various workloads, providing critical metrics like latency and throughput to ensure the model meets performance requirements before deployment.

How it works

The tool uses a scalable multiprocess architecture (10 services communicating via ZMQ) to simulate user traffic against an inference endpoint. It supports multiple load control modes—including concurrency, request rate, and deterministic trace replay—and can be integrated with telemetry tools like MLflow, OpenTelemetry, and Weights & Biases for real-time monitoring. It is extensible via a plugin system for endpoints, datasets, and metrics.

Who it’s for

It is built for AI engineers, ML Ops professionals, and researchers who need to rigorously test the performance of LLMs, vision, audio, and video models across different inference engines (such as vLLM, SGLang, or TensorRT-LLM) and hosting environments (including Kubernetes).

Highlights

• Comprehensive Metrics: Tracks Time to First Token (TTFT), Inter Token Latency (ITL), and request throughput.
• Diverse Workload Support: Supports public datasets (ShareGPT), custom prompts, and production trace replay for realistic traffic simulation.
• Broad Modality Support: Benchmarks text, image generation, embeddings, audio, and video models.
• Advanced Load Control: Features adaptive scaling for SLA boundary discovery, gradual ramping, and warmup phases to eliminate cold-start effects.
• Cloud Native: Includes dedicated Kubernetes support with an operator and Helm charts for cluster-scale benchmarking.