VAKRA Benchmark Analysis: Agent Reasoning, Tool Use, and Failure Modes

TL;DR

Hugging Face released the VAKRA benchmark, an executable suite of over 8,000 locally hosted APIs across 62 domains that tests AI agents on compositional reasoning, tool selection, multi‑hop workflows, and policy compliance; current state‑of‑the‑art models perform poorly, exposing critical gaps for real‑world deployment.

What Is VAKRA?

VAKRA is a tool‑grounded, executable benchmark designed to evaluate how well AI agents can reason and act in enterprise‑like environments. Unlike traditional isolated‑skill tests, VAKRA measures compositional reasoning across APIs and documents by requiring agents to execute full multi‑step workflows and providing execution traces for verification.

Key statistics:

  • 8,000+ locally hosted APIs backed by real databases.
  • 62 domains covering business intelligence, dashboards, and document retrieval.
  • Tasks involve 3–7 step reasoning chains that mix structured API calls with unstructured document retrieval.
  • Four capability groups test distinct skill sets (API chaining, tool selection, multi‑hop reasoning, and multi‑source reasoning with policy constraints).

The benchmark, dataset, leaderboard, and code are publicly available on Hugging Face and GitHub.

Capability 1 – API Chaining with Business‑Intelligence APIs

  • Instances: 2,077 across 54 domains.
  • Tool collections: SLOT‑BIRD (7 generic tools) and SEL‑BIRD (expanded, domain‑specific getters).
  • Workflow: Begins with get_data(tool_universe_id) to load a lightweight preview and configure the server, followed by 1–12 chained tool calls to filter and select data.
  • Example JSON request (simplified):
    {
  "query": "Which football team has a build‑up play speed of 31 …?",
  "tool_calls": [
    {"name": "get_data", "arguments": {"tool_universe_id": "..."}, "label": "retrieved_data_1"},
    {"name": "select_data_equal_to", "arguments": {"data_label": "retrieved_data_1", "key_name": "play_speed", "value": 31}, "label": "FILTERED_DF_0"},
    ...
  ],
  "answer": "FC Barcelona"
}
  • Challenge: Selecting the correct tool from a large, dynamic set and supplying many optional parameters.

Capability 2 – Tool Selection with Dashboard APIs

  • Instances: 1,597 across 17 domains.
  • Tool collection: REST‑BIRD, served via FastAPI and wrapped by the MCP server.
  • Domain size: 6–328 tools per domain (average 116).
  • Constraint: OpenAI’s tool‑list limit of 128 items forces agents to implement a shortlisting mechanism; the baseline uses a simple heuristic shortlister.
  • Goal: Identify the single correct endpoint‑style API for a query.

Capability 3 – Multi‑Hop Reasoning with Dashboard APIs

  • Instances: 869 across 38 domains.
  • Requirement: 1–5 logical hops; each hop must invoke the correct API and pass appropriate arguments.
  • Observation: Accuracy drops sharply as hop depth increases, confirming that chaining multiple API calls is a major difficulty for current models.

Capability 4 – Multi‑Hop, Multi‑Source Reasoning & Policy Adherence

  • Instances: 644 across 41 domains.
  • Features:
    • Multi‑Source: Queries may require a sequence such as API → Document Retrieval (RAG) → API, with source de‑contamination ensuring each hop’s answer is available from only one source.
    • Multi‑Turn: Dialog context is provided; the agent must answer the current turn only.
    • Tool‑Usage Policies: Plain‑text constraints dictate which knowledge sources may be used (e.g., “only use document retrievers for technology queries”).
  • Policy enforcement: Baseline agents prepend a constraint sentence to the prompt; developers may implement more sophisticated checks.

Evaluation Framework

VAKRA uses an execution‑centric, waterfall‑style pipeline that validates agents on both tool‑execution correctness and final answer quality.

  1. Policy adherence (Capability 4 only) – programmatically verified.
  2. Tool‑call sequence validation – predicted calls are executed; responses are compared to ground‑truth responses.
    • If exact match fails, an LLM‑based evaluator (adapted from CRAG) checks whether the predicted trajectory retrieves all required information, allowing alternative but valid tool sequences.
  3. Final response evaluation – an LLM judge confirms that the answer is grounded in the executed tool outputs and factually matches the reference.

Scoring

All four capabilities receive equal weight:

$$\text{Leaderboard Score}=\frac{1}{4}\sum_{i=1}^{4}\text{Capability}_i$$

  • Capabilities 1‑3: Simple accuracy (correct queries / total queries).
  • Capability 4: Multi‑source queries are weighted twice to reflect higher difficulty.

Error Analysis – Where Agents Break

The analysis categorizes failures by the first point of breakdown:

  1. Wrong tool selection.
  2. Missing or hallucinated arguments.
  3. Incorrect argument values.
  4. Incorrect or ungrounded final response.

API Chaining (Capability 1)

  • Best model: GPT‑OSS‑120B, primarily due to superior understanding of tool schemas and robust handling of optional parameters.
  • Error patterns: Models using the SLOT‑BIRD collection struggled with argument naming; those using SEL‑BIRD made more tool‑selection mistakes because of the larger tool set.

Dashboard Tool Selection (Capability 2)

  • Best model: Gemini‑3‑flash‑preview, outperforming others across all error categories.
  • Dominant errors: Tool‑selection failures and incorrect parameter values; synthesis of the final answer remained a challenge even when tool calls succeeded.

Multi‑Hop Reasoning (Capability 3)

  • Trend: Accuracy declines with hop depth (1‑hop > 2‑hop > 3+‑hop). All models show the same pattern, confirming that chaining multiple API calls amplifies error propagation.

Multi‑Hop Multi‑Source & Policies (Capability 4)

  • Hybrid hops: Instances requiring both API calls and document retrieval are hardest; performance drops sharply compared to pure API or pure RAG hops.
  • Policy impact: Models generally obey policies poorly; most experience a noticeable accuracy drop when policies restrict the most relevant source. Granite‑4.0‑h‑Small‑32B is an exception, showing less degradation.
  • Specific observations: GPT‑OSS‑120B often skips a single‑hop RAG call, answering from internal knowledge instead; Gemini‑3‑flash‑preview excels on 2‑hop API‑RAG combos, likely leveraging its strength on dashboard APIs.

Implications for Agent Development

  • Tool competence ≠ reliability: Selecting the right API is only part of the problem; agents must also manage arguments, execute multi‑step workflows, and respect external constraints.
  • Execution‑centric metrics matter: Traditional answer‑only scores hide failures in intermediate steps; VAKRA’s trajectory‑based evaluation surfaces these gaps.
  • Policy handling is a weak spot: Real‑world deployments often impose compliance or security policies; current models struggle to incorporate such constraints into their reasoning.
  • Benchmark as a development target: By exposing where agents break, VAKRA provides a concrete roadmap for improving tool‑use modules, shortlisting mechanisms, and policy‑aware prompting.

How to Try VAKRA

Run your agent against the benchmark to discover whether it fails on tool selection, multi‑hop reasoning, or policy compliance, and iterate based on the detailed error breakdown provided by VAKRA.

Sources