Tool‑Eval‑Bench – A benchmark for LLM‑tool‑calling

---

What & Why

Tool‑Eval‑Bench is a benchmark suite that measures how well large language models (LLMs) can call external tools in a single‑assistant‑conversation workflow. It is aimed at developers who run self‑hosted inference servers (vLLM, llama.cpp, SGLang, LiteLLM, TensorFold, etc.) or hosted APIs such as Gemini or Anthropic and want a reproducible way to compare tool‑selection, parameter passing, chaining, error‑recovery, safety and localisation performance.

Core features

Feature	What you get	Where it lives
Quality benchmark	69 deterministic scenarios (plus 28 optional Hard‑Mode adversarial cases) that cover 52 tool namespaces, multi‑step chains, refusals, safety checks, localisation (German‑focused) and more.	run command
Throughput & speed	Pre‑fill and generation speed tests (llama‑bench style) with depth‑ and concurrency‑sweeps; speculative‑decoding metrics.	bench command
Long‑context retrieval	Needle‑in‑a‑haystack tests across many context lengths, reporting effective context.	needle flag
Context‑pressure	Vary the amount of pre‑filled context before each scenario to see where quality drops.	context‑pressure flag
Accuracy suite	GSM8K, MMLU and IFEval run through the same OpenAI‑compatible adapter.	plugin command
Decision‑model audit	Optional secondary model can audit the assistant’s answer (probabilities, agreement).	--decision‑judge flag
Persisted artifacts	Markdown report with full conversation trace + SQLite DB for querying.	runs/…/*.md & data/benchmarks.sqlite
CLI & API	Full command‑line interface; Python async API (run_benchmark).	tool‑eval‑bench executable & tool_eval_bench.api
Backend autodetection	Detects vLLM, llama.cpp, TensorFold, Strata, TabbyAPI, etc., via headers, /health, /props, etc.	Automatic during run/probe
Config fingerprint	Captures version, model, context window, quantisation, GPU count, speculative mode, etc., to ensure fair leaderboard comparisons.	Stored in run metadata

How to use it (quick start)

# Install via uv (or Docker)
uv tool install git+https://github.com/SeraphimSerapis/tool-eval-bench.git
# Run the short 15‑scenario suite against a local server
tool-eval-bench run --short --base-url http://localhost:8000
# Full benchmark (69 scenarios) – reproducible seed
tool-eval-bench run --seed 42
# Add Hard‑Mode and performance tests
tool-eval-bench run --hardmode --perf --seed 42
# View past runs or leaderboard
tool-eval-bench history
tool-eval-bench leaderboard

Environment variables (TOOL_EVAL_*) or a .env file let you store URLs, API keys and model names for repeated A/B testing.

What the scores mean

• Each scenario scores 2 (pass), 1 (partial) or 0 (fail). The composite score is the percentage of points earned.
• Rating stars:
  • 90‑100 % → ★★★★★ Excellent
  • 75‑89 % → ★★★★ Good
  • 60‑74 % → ★★★ Adequate
  • 40‑59 % → ★★ Weak
  • 0‑39 % → ★ Poor
• Safety violations can cap the rating at ★★★ regardless of raw score.
• --weight-by-difficulty re‑weights harder scenarios for an alternative score.

When it’s useful

• Model developers who need a concrete, repeatable metric for tool‑calling ability before releasing a new checkpoint.
• Ops teams comparing self‑hosted back‑ends (vLLM vs. llama.cpp vs. TensorFold, etc.) under identical workloads.
• Research on prompting strategies, system‑prompt tweaks, or speculative decoding impact on tool use.
• CI pipelines – you can fail a build on a safety regression (--fail‑on‑safety).

Limitations & scope

• Focuses on a single assistant conversation with mock tools; it does not evaluate full multi‑agent systems, delegation, or long‑running autonomous agents.
• Current localisation is German‑centric; other languages are not covered out‑of‑the‑box.
• Difficulty tiers are author estimates, not calibrated against a large model zoo.
• Mock tool responses include realistic noise, but they are still simulated—not live external APIs.

---

All information above is taken directly from the repository’s README; no external assumptions have been added.