Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents

Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents

Overview

VAKRA is a tool‑grounded, executable benchmark that measures how well AI agents reason and act in enterprise‑like environments by requiring compositional reasoning across APIs and documents. It matters because it exposes a gap between surface‑level tool competence and reliable end‑to‑end agent behavior, showing that even strong models often fail on multi‑step workflows.

Task Description

VAKRA comprises four capabilities, each testing a distinct set of skills in an executable environment with over 8,000 locally hosted APIs across 62 domains and domain‑aligned document collections.

Capability 1: API Chaining using Business Intelligence APIs

This capability contains 2,077 test instances across 54 domains, requiring agents to chain 1–12 tool calls from the SLOT‑BIRD and SEL‑BIRD collections to derive answers from JSON data sources. Each instance begins with a get_data(tool_universe_id=id) call that initializes the data source and configures the MCP server to expose the appropriate tool set.

Capability 2: Tool Selection using Dashboard APIs

This capability includes 1,597 instances across 17 domains, using the expanded REST‑BIRD collection of endpoint‑style APIs served via FastAPI wrapped by the MCP server. Agents must select the correct APIs from a domain‑specific set that ranges from 6 to 328 tools per domain (average 116). The OpenAI API Specification limits the tool list to 128 tools, necessitating a shortlisting mechanism.

Capability 3: Multi‑Hop Reasoning using Dashboard APIs

The Capability 3 segment has 869 test instances drawn from 38 subject domains, again relying on the REST‑BIRD API collection but adding multi‑hop reasoning. Questions require between one and five logical hops, where each hop involves extracting and combining supporting evidence from API calls.

Capability 4: Multi‑Hop, Multi‑Source Reasoning and Policy Adherence

Capability 4 includes 644 instances across 41 domains, built on the REST‑BIRD API collection with added document indices per domain, multi‑turn conversations, and optional tool‑usage policies. Queries may require information from APIs, document retrievers, or a combination (API‑RAG‑API patterns). Policies are plain‑text instructions that constrain which knowledge sources an agent may use for a given turn.

Evaluation Framework

VAKRA evaluates agents by assessing both the correctness of the final answer and the validity of the full tool‑execution trajectory, rewarding agents that obtain correct answers through valid reasoning processes.

Evaluation Metric

The VAKRA Evaluator operates over a predicted final response and the corresponding tool‑call trajectory, executing predicted calls in the same environment as the ground truth to verify intermediate outputs. Evaluation follows a waterfall‑style pipeline: for Capability 4 tasks, policy adherence is checked first; then the predicted tool‑call sequence is compared against the ground truth; only samples with valid trajectories proceed to final response evaluation.

Tool‑sequence correctness is determined by executing each predicted tool and comparing the set of tool responses to those from the ground truth, allowing alternative but valid invocations. If the check is inconclusive, an LLM‑based evaluation adapted from the CRAG framework determines whether the predicted trajectory retrieves all required information despite structural differences. Final responses are judged by an LLM to ensure they are grounded in the predicted tool outputs and factually consistent with the ground truth answer.

Scores are computed per capability and averaged for the leaderboard: Leaderboard Score = (Capability₁ + Capability₂ + Capability₃ + Capability₄) / 4. Capabilities 1‑3 are simple averages of correct queries; Capability 4 weights multi‑source correct queries twice as much as API‑only or RAG‑only correct queries.

Error Analysis

Error analysis uses stage‑wise categorization, assigning each failure to the first point of breakdown: tool selection, argument provision, argument values, or final response grounding.

Failure Stage Isolation

By classifying each instance to the earliest failing stage, errors are treated as disjoint fractions of the dataset, avoiding double‑counting and providing an interpretable breakdown.

Capability 1: API Chaining using Business Intelligence APIs

For the 2,077 samples in this capability, GPT‑OSS‑120B outperformed other models by a large margin, mainly due to better understanding of tool schemas and robustness in filling optional parameters. Errors with SLOT‑BIRD (1,477 samples) were dominated by incorrect tool argument names for all models except GPT‑OSS‑120B, while SEL‑BIRD (600 samples) showed fewer argument errors but more tool‑selection errors due to a larger, dynamic tool set.

Capability 2: Tool Selection using Dashboard APIs

Across the 1,597 samples, Gemini‑3‑flash‑preview outperformed other models on all error categories. The large number of tool options leads to frequent errors in tool selection and parameter value selection, while hallucinating or skipping required parameters is rare. Even when all tool calls are correct, models (especially Gemini‑3‑flash‑preview and Claude‑Sonnet‑4‑5) struggle to synthesize a correct answer from tool responses, evidenced by drop‑offs at the far right of the error plot.

Multi‑Hop Reasoning: Effect of Hop Depth on Model Performance

Accuracy declines as hop depth increases: models perform best on single‑hop questions, with degraded performance on two‑hop and further degradation on three‑or‑more‑hop questions across all tested models.

Multi‑Hop Multi‑Source Reasoning: Effect of Hybrid Hops on Model Performance

Performance varies by interaction type: single API calls (1‑hop API) are easier than multiple API invocations (2‑hop API); adding document retrievers (RAG hops or hybrid API‑RAG patterns) increases difficulty. On 1‑hop RAG questions, GPT‑OSS‑120B tends to return answers from parametric knowledge instead of invoking the retriever, likely because the questions are Wikipedia‑entity focused. Gemini‑3‑flash‑preview shows relatively strong performance on 2‑hop API‑RAG patterns, likely benefiting from its dashboard‑API tool‑selection strength.

Effect of Policies on Model Performance

When policies restrict access to the most relevant information source ("Policy updates the answer)), all models except Granite‑4.0‑h‑Small‑32B experience a clear performance drop. Models either violate constraints or fail to retrieve sufficient information, sometimes understanding the policy but still answering incorrectly. This indicates that while models can reason over tools and sources, they struggle to incorporate external constraints into that reasoning—a key requirement for reliable real‑world deployment.

Conclusion

VAKRA exposes a critical gap between surface‑level tool competence and robust, end‑to‑end agent reliability. Although modern models can increasingly select APIs and execute isolated tool calls, the benchmark shows these abilities alone are insufficient for real‑world deployment; models often break down when required to perform compositional reasoning under execution constraints spanning APIs, documents, dialog context, and policy requirements.

Try VAKRA — Where Does Your Agent Break?

Test your agent on the VAKRA benchmark to see where it fails—tool selection, multi‑hop reasoning, or policy constraints.

👉 Try it and tell us what your agent learned

Sources