SeraphimSerapis/tool-eval-bench

What it solves

It provides a standardized way to measure the tool-calling quality of Large Language Models (LLMs) specifically when they are deployed on self-hosted serving stacks. It helps developers determine if a model can accurately select tools, provide precise parameters, and handle complex agentic workflows without needing to build a full agentic system for testing.

How it works

The benchmark runs a series of deterministic scenarios (69 standard and 19 "Hard Mode" cases) through an OpenAI-compatible API endpoint. It uses mock tools with realistic, noisy payloads to simulate real-world APIs. The system scores each scenario as pass, partial, or fail, then aggregates these into a composite score and a star rating. Beyond tool-calling, it can also measure throughput, long-context retrieval (needle-in-a-haystack), and general accuracy (GSM8K, MMLU).

Who it’s for

It is designed for developers and researchers using self-hosted inference engines like vLLM, SGLang, llama.cpp, and LiteLLM who need to validate the tool-calling capabilities and performance of their models.

Highlights

• Comprehensive Tool Testing: Covers tool selection, multi-step chains, error recovery, safety/prompt injection, and structured output.
• Hard Mode: Includes adversarial and stateful scenarios for high-performing models.
• Performance Suite: Integrated benchmarking for throughput, speculative decoding, and context pressure.
• Detailed Reporting: Generates Markdown reports with full conversation traces and a queryable SQLite database of results.
• Broad Backend Support: Works with any OpenAI-compatible /v1/chat/completions endpoint.