AISBench/benchmark
AISBench Benchmark is a model evaluation tool built on OpenCompass, compatible with OpenCompass’s configuration system, dataset structure, and model backend implementation, while extending support for service-based models.
AISBench Benchmark 是一个面向 AI 模型的评测工具，基于 OpenCompass 构建，兼容其配置体系、数据集结构与模型后端，并额外扩展了对服务化模型（如 vLLM、Triton）的支持。
它支持两大类评测：
- 精度评测：对服务化或本地模型在问答、推理、多模态等基准数据集（如 GSM8K、MMMU、SWE-Bench）上的准确率进行评估。
- 性能评测：评估服务化模型的延迟、吞吐量，支持高并发、稳态性能、真实流量模拟、prefix cache 命中率测试等。
主要特性：
- 多任务并行：可同时运行多个精度或性能评测任务，提升效率。
- 多模态支持：支持图像、视频生成质量评估（如 VBench、OneIG-Benchmark）。
- 智能体评测：对接 SWE-Bench、τ²-Bench 等智能体基准，支持 Docker 容器内运行。
- 数据集丰富：内置 100+ 开源数据集，并支持自定义数据集。
- 灵活配置：提供 Python 配置文件（推荐）或命令行参数两种方式。
安装与使用：
- 支持 Python 3.10/3.11/3.12，可通过 pip 安装（
pip install ais_bench_benchmark）或源码安装。
- 需要先启动一个 OpenAI 兼容的推理服务（如 vLLM），指定模型服务地址、数据集，运行
ais_bench 命令即可开始评测。
项目文档完善（ReadTheDocs），提供中文与英文版。整体定位是面向生产环境的 AI 模型评测工具，适合需要标准化评测流程的团队。
Related
- Project
modelscope/evalscopeA comprehensive LLM evaluation framework that combines capability benchmarking, inference performance stress testing, and interactive result visualization in one tool.
- Project
ai-dynamo/aiperfA comprehensive benchmarking tool for generative AI models that measures latency and throughput across various inference solutions and modalities.
- Project
IBM/AssetOpsBenchAssetOpsBench is an open‑source benchmark/framework for building, orchestrating, and evaluating LLM‑based AI agents that operate on industrial asset‑management data (sensors, work orders, failure modes, etc.). It provides domain‑specific MCP tool servers, several ReAct‑style agent back‑ends, 141+ realistic scenarios, and a multi‑dimensional evaluation pipeline used in KDD/AAAI/NeurIPS competitions.
- Project
MatthewCYM/VoiceBenchVoiceBench is a benchmark suite for evaluating LLM‑based voice assistants. It supplies a Hugging‑Face dataset with audio‑driven QA and instruction‑following tasks, a command‑line toolkit to run a model, collect spoken responses, and automatically score them (via GPT‑4o‑mini for open‑ended sets). Results are posted on a public leaderboard, and the repo includes a curated list of recent voice‑assistant papers.
- Project
Tencent/AI-Infra-GuardAI‑Infra‑Guard (A.I.G) is an open‑source, Docker‑deployable platform for red‑team security testing of AI services. It scans live LLM endpoints, AI‑agent workflows, and model‑control‑plane code for CVEs, jailbreak susceptibility, and agent loss‑of‑control risks, exposing results via a web UI and REST API.