SeraphimSerapis/tool-eval-bench

Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, and structured output. Supports vLLM, SGLang, and llama.cpp.

What it solves

It provides a standardized way to measure the tool-calling quality of Large Language Models (LLMs) specifically when they are deployed on self-hosted serving stacks. It helps developers determine if a model can accurately select tools, provide precise parameters, and handle complex agentic workflows without needing to build a full agentic system for testing.

How it works

The benchmark runs a series of deterministic scenarios (69 standard and 19 "Hard Mode" cases) through an OpenAI-compatible API endpoint. It uses mock tools with realistic, noisy payloads to simulate real-world APIs. The system scores each scenario as pass, partial, or fail, then aggregates these into a composite score and a star rating. Beyond tool-calling, it can also measure throughput, long-context retrieval (needle-in-a-haystack), and general accuracy (GSM8K, MMLU).

Who it’s for

It is designed for developers and researchers using self-hosted inference engines like vLLM, SGLang, llama.cpp, and LiteLLM who need to validate the tool-calling capabilities and performance of their models.

Highlights

  • Comprehensive Tool Testing: Covers tool selection, multi-step chains, error recovery, safety/prompt injection, and structured output.
  • Hard Mode: Includes adversarial and stateful scenarios for high-performing models.
  • Performance Suite: Integrated benchmarking for throughput, speculative decoding, and context pressure.
  • Detailed Reporting: Generates Markdown reports with full conversation traces and a queryable SQLite database of results.
  • Broad Backend Support: Works with any OpenAI-compatible /v1/chat/completions endpoint.

Related

  • Project
    vllm-project/guidellm

    An SLO-aware benchmarking and evaluation platform that simulates real-world workloads to optimize LLM inference performance, resource needs, and operational limits.

  • Project
    aidatatools/ollama-benchmark

    A cross-platform CLI tool that measures the tokens-per-second throughput of local LLMs running via Ollama based on the user's available system RAM.

  • Project
    confident-ai/deepeval

    An open-source LLM evaluation framework that provides unit-testing capabilities for AI agents, RAG pipelines, and chatbots using LLM-as-a-judge metrics.

  • Project
    hkust-nlp/Toolathlon

    Toolathlon is a benchmark for assessing language agents' general tool-use capabilities across 600+ diverse tools in realistic, long-horizon software environments.

  • Project
    sgl-project/genai-bench

    A performance benchmarking tool for LLM serving systems that provides token-level metrics, real-time monitoring, and detailed visual reports.