DABStep: Data Agent Benchmark for Multi-step Reasoning

Hugging Face and Adyen have developed the Data Agent Benchmark for Multi-step Reasoning (DABStep), a specialized evaluation framework consisting of over 450 data analysis tasks. The benchmark reveals a significant performance gap in current AI agents, with the most capable reasoning-based models achieving only 16% accuracy on the hard task set.

DABStep Benchmark Design

DABStep is designed to evaluate agentic workflows—where LLMs use tools to independently execute multi-step tasks—specifically within the context of real-world data analysis. Unlike synthetic benchmarks, DABStep uses tasks extracted from actual Adyen workloads to simulate the challenges faced by professional data analysts.

Key Characteristics

  • Real-World Use Cases: The benchmark utilizes over 450 tasks based on actual workloads, avoiding "toy problems" to better reflect daily analytical challenges.
  • Structured and Unstructured Data: Tasks require models to navigate both structured datasets (CSV, JSON) and unstructured domain knowledge (markdown manuals, handbooks).
  • Factoid Evaluation: To ensure objective, model-free grading, the benchmark uses binary (right/wrong) outcomes based on simple words, numbers, or multiple-choice answers.
  • Multi-step Complexity: Tasks are designed so they cannot be solved with a single shot of code; they require iterative problem-solving and sequential reasoning.

Data Composition

The benchmark includes various financial payments sector datasets, such as:

  • payments.csv: Anonymized transactions with fraud and risk signals.
  • fees.json: A dataset of 1,000 Scheme Fee structures.
  • manual.md: A distilled handbook of essential business knowledge for solving tasks.
  • Other metadata: Tables for acquirer countries and merchant category codes (MCCs).

Task Difficulty and Generalization

DABStep categorizes tasks into two difficulty levels to provide a granular view of agent capabilities:

  • Easy Level: These serve as warm-ups requiring a single structured dataset and minimal context. Llama 70B zero-shot prompts can exceed 90% accuracy on these tasks.
  • Hard Level: These require multiple structured datasets and domain-specific knowledge. These tasks necessitate multi-step inductive reasoning and iterative code generation.

To prevent "lucky guesses" and ensure core reasoning repeatability, the benchmark employs symbolic reasoning by exploding task cardinality through permutations of time ranges and merchant names. It also utilizes a heldout test set to evaluate how well agents generalize across analysis tasks not captured in the initial version.

Performance Baselines and Model Analysis

Initial evaluations on the Hard set show that even the most advanced reasoning models struggle with complex data analysis.

Accuracy Results

  • o3-mini: 16% accuracy (top performer).
  • DeepSeek R1: 13% accuracy.
  • Claude 3.5 Sonnet: 12% accuracy.
  • DeepSeek V3: 6% accuracy.

Technical Observations

Researchers found that reasoning models (like o1 and R1) performed poorly (0% accuracy) when using a standard ReAct prompt that worked well for chat models. These models required a specialized "Reasoning Prompt" to function effectively. Common failure modes included invalid code syntax, poor instruction following, and a failure to execute sequential steps.

Cost-Performance Tradeoff

Analysis of commercial offerings shows varying economics for these agents:

  • o1 was the most expensive at $435 for the full benchmark ($0.967 per task).
  • DeepSeek R1 and GPT-4o-mini were the most cost-effective at $3 for the full benchmark ($0.007 per task).
  • The researchers noted that DeepSeek R1 offers an ideal economic profile with high performance relative to its low cost.

Future Roadmap for DABStep

Hugging Face and Adyen plan to evolve the benchmark to maintain its validity as AI models improve. Future expansions include:

  • Expanded Task Scope: Moving beyond fraud and fees into approval rates, authentication drop-offs, and seasonal components.
  • Cross-Domain Application: Incorporating datasets from health, biology, insurance, and telecommunications.
  • Increased Data Scale: Introducing datasets that exceed memory limits, requiring the use of distributed computing engines.
  • Complex Documentation: Adding PDF formats and versioned logic to test context window limits.
  • Multimodal Capabilities: Adding tasks that require interpreting and creating plots and graphs.

Sources