ThinkingBox Benchmark Reveals Consistency Gaps in AI Agents Across 507 Business Workflows

TL;DR

ThinkingBox, now available through Hugging Face, evaluates AI agents by checking the final database state they leave behind, exposing that most agents succeed on a single run but fail to repeat the correct outcome across 20 attempts, with significant cost‑consistency trade‑offs among 12 LLMs.

A tool call is not an outcome

Agents can issue valid tool calls yet still produce the wrong database record.

  • In a 121,680‑trial ablation across 12 LLMs, 79,853 attempts failed executable checks.
  • 67.24 % of those failures terminated cleanly, reported no tool error, and still changed state.
  • Among the failed traces, 77.61 % had wrong field values, 43.30 % produced unintended side effects, and 25.36 % missed required effects.

"A trajectory is a claim. Database state is the evidence. Repetition is the trust test."

The benchmark therefore grades agents on the terminal backend state rather than on the surface dialogue or tool‑call count.

One success is not reliability

Running each task 20 times reveals how often an agent can be trusted.

Three metrics are reported:

Metric What it measures What it answers
pass@1 Share of all attempts that succeeded How does the model usually perform?
pass@20 Share of tasks solved at least once in 20 tries Can it ever solve the task?
Observed 20/20 Tasks that succeeded in all 20 attempts Can it always be correct?

The blog focuses on Observed 20/20 – the literal count of tasks that passed every repeat.

Can you depend on the model behind your agent?

Breadth (pass@20) and consistency (Observed 20/20) diverge across models.

  • Kimi‑K3 solves 93.89 % of tasks at least once (largest breadth) but achieves only 13.41 % observed 20/20.
  • Claude Opus 5 solves fewer tasks at least once (79.09 %) but attains 47.53 % observed 20/20, the highest consistency.
  • Claude Opus 5.5 improves pass@1 (67.16 % vs 66.50 %) but does not increase the number of tasks that are consistently correct (still 241 observed 20/20).

"If you are choosing a model for work that touches real records, pass@20 is the wrong column to look at."

What consistency costs

Cost per successful task is computed from token usage priced at OpenRouter list rates.

  • GPT‑5.4: $43.49 for 507 attempts → $0.131 per successful task.
  • GPT‑5.6 Sol: lowest cost per success at $0.127.
  • Claude Opus 5.5: $0.276 per success, higher accuracy.

Pareto cost frontier

Three models lie on the frontier (no other model is both cheaper and more accurate):

  1. GPT‑5.6 Sol – cheapest per success.
  2. GPT‑5.4 – modest accuracy boost for $0.004 more per success.
  3. Claude Opus 5.5 – further 1.8 % accuracy gain for $0.276 per success.

Cost per dependable task (20‑run consistency)

Model Tasks passing 20/20 Estimated cost (20 runs) Cost per dependable task
GPT‑5.4 128 (25.25 %) $869.80 $6.80
GPT‑6 Astra 231 (45.56 %) $1,720.60 $7.45
Claude Opus 5.5 241 (47.53 %) $1,880.77 $7.80
… … … …

The cheapest model for a single correct answer is not the cheapest for a consistently correct answer.

Failure signatures

Four diagnostic categories explain most failures.

Signature Share of failures
Tool usage 79.9 %
Wrong state updates 10.3 %
Incomplete user resolutions 7.0 %
No state‑changing action 2.9 %

The dominant issue is tool‑handling errors rather than model reasoning. Agents often reach the workflow stage where a tool call fails and then cannot recover, indicating a need for better retry and error‑recovery logic.

How it works

ThinkingBox provides a sandbox environment and a benchmark dataset (ThinkingBox‑Bench).

  1. Task definition – initial backend state, user goal, available MCP tools, policy, and executable checks.
  2. Isolated MCP session – each of the 20 repeats starts from a fresh database snapshot.
  3. Agent execution – the model interacts with tools, updates state, and produces a final response.
  4. Side‑effect extraction – records actual database changes.
  5. Deterministic judges – compare extracted state against required end state; any trajectory achieving the correct state passes.

All evaluation logic runs on the evaluator side; the model never sees the golden state or grading code.

Run it yourself

ThinkingBox is released on Hugging Face with full OpenEnv integration.

Prerequisites

  • Linux/WSL, Python 3.11+, uv, Docker.
  • Clone the pinned releases of OpenEnv, thinkingbox-data, and install the thinkingbox CLI.

Quick start (excerpt)

# Clone and install OpenEnv + ThinkingBox env
git clone https://github.com/huggingface/OpenEnv && cd OpenEnv
uv sync --project envs/thinkingbox_env --frozen

# Clone data repo at v1.0
git clone https://github.com/microsoft/thinkingbox-data && \
  git -C thinkingbox-data checkout thinkingbox-bench-v1.0

# Install CLI
uv tool install "thinkingbox @ git+https://github.com/microsoft/thinkingbox"

Then start Typesense, the MCP Session Proxy, and the OpenEnv server as described in the blog (three separate terminals). After readiness checks, evaluate a single task:

echo "- sandbox_external_retail_group1.py:test_case_ST002_001" > one_task.yaml
uv run --project envs/thinkingbox_env thinkingbox-eval \
  one_task.yaml \
  --config "$PWD/thinkingbox.yaml" \
  --output results.jsonl \
  --repeat 1 --message-timeout 1800

The CLI returns a binary pass/fail reward and logs any system errors for reproducibility.

Where this goes next

The benchmark’s primary contribution is the environment that makes repeatability and state‑based grading easy to adopt.

Recommendations for practitioners:

  1. Inspect failures – look at the actual database changes, not just the final dialogue.
  2. Reproduce tasks with your own model via OpenEnv.
  3. Report a repeat metric (e.g., best‑of‑k or every‑of‑k) and be explicit about the aggregation method.

All code is open‑source (MIT for the sandbox, CDLA‑Permissive‑2.0 for the data). The benchmark remains synthetic but mirrors real‑world enterprise agent patterns.

Resources

  • ThinkingBox environment: https://huggingface.co/docs/openenv/environments/thinkingbox
  • ThinkingBox‑Bench dataset: https://huggingface.co/datasets/microsoft/ThinkingBox-Bench
  • Paper: arXiv:2608.19741
  • GitHub repos: microsoft/thinkingbox, microsoft/thinkingbox-data, microsoft/thinkingbox-training (RL training forthcoming)

Sources