ag2ai/Agents_Failure_Attribution

Benchmark for automated failure attributions in agentic systems (🏆 ICML 2025 Spotlight)

What it solves

Manually debugging failures in LLM-based multi-agent systems is time-consuming. This project automates failure attribution: given a failed task, it identifies which agent caused the failure and at which step, so developers can quickly find and fix the root cause instead of tracing logs by hand.

How it works

The repository provides a benchmark dataset called Who&When, containing 184 annotated failure trajectories from both algorithm-generated and hand-crafted multi-agent systems. Each failure includes the responsible agent, the decisive error step, and a natural-language explanation. The code supports three attribution methods — all-at-once judging, step-by-step judging, and binary search — and works with several models like GPT-4o and Llama-3.1. You run an inference script on a dataset, then evaluate the outputs with a separate script.

Who it’s for

Developers and researchers working on LLM-based multi-agent systems who need to debug failures, improve agent performance, or benchmark failure-attribution techniques. It’s also useful for anyone studying how to make agentic systems self-correct using pinpointed feedback.

Highlights

  • First benchmark specifically for automated failure attribution in multi-agent LLM systems.
  • 184 annotated failure tasks with fine-grained labels: responsible agent, decisive step, and explanation.
  • Covers both hand-crafted and algorithm-generated agentic systems, based on GAIA and AssistantBench queries.
  • Supports multiple models (GPT-4o, Llama, Qwen, etc.) and three inference strategies (all-at-once, step-by-step, binary search).
  • Open-sourced with inference and evaluation scripts, along with a reference paper and dataset on Hugging Face.

Related

  • Project
  • Project
  • Project
  • Project