AISBench/benchmark

What it solves

AISBench Benchmark is a comprehensive evaluation tool designed to measure the performance and accuracy of AI models, with a specific focus on extending support for "service-based" (API-driven) models like those deployed via vLLM or Triton. It addresses the need for a standardized way to validate model quality across various modalities and stress-test inference services under realistic workloads.

How it works

Built upon the OpenCompass framework, AISBench integrates with its configuration system and dataset structures. It operates across two primary evaluation modes:

1. Accuracy Evaluation: Validates the correctness of responses from both local and service-based models using a wide array of question-answering and reasoning benchmarks (covering text and multimodal data).
2. Performance Evaluation: Measures latency and throughput of service-based models, including extreme stress testing, steady-state performance analysis, and simulation of real-world business traffic patterns.

Users can configure tasks via Python scripts or command-line arguments, specifying the model backend, the dataset, and the desired summarization method.

Who it’s for

• AI Engineers and Researchers who need to benchmark the accuracy of their models against industry standards.
• MLOps Engineers tasked with optimizing the throughput and latency of inference services.
• Developers deploying LLMs or multimodal models who need to verify service stability and performance under high concurrency.

Highlights

• Broad Benchmark Support: Integrates numerous benchmarks including SWE-Bench, TAU2-Bench, VBench, and OneIG-Benchmark.
• Multimodal Capabilities: Supports evaluation of text, image generation (GEdit-Bench), and video generation (VBench).
• Service-Oriented: Specialized support for API-based models with features like request rate control and concurrency management.
• Advanced Performance Tools: Includes real-world traffic simulation and steady-state performance testing to find the true optimal performance of a system.
• Flexible Deployment: Provided as a Python package via PyPI and as multi-architecture Docker images (x86_64 and ARM).