IBM and UC Berkeley Diagnose Enterprise Agent Failures Using IT-Bench and MAST

IBM Research and UC Berkeley have developed a method to move beyond simple success rates in agentic LLM evaluations by combining IT-Bench—a benchmark for SRE, Security, and FinOps automation—with MAST (Multi-Agent System Failure Taxonomy). This approach allows developers to identify specific failure signatures, distinguishing between benign structural frictions and fatal errors that lead to task collapse.

The "Black Box" Problem of Agent Benchmarks

Traditional benchmarks typically report a single success rate, which fails to explain why an agent failed. To solve this, the researchers introduced MAST, a standardized taxonomy that converts unstructured execution logs into structured "failure vectors" across 14 distinct patterns. These patterns are grouped into three categories:

  • FC1: System Design Issues (The Skeleton): Architectural failures such as FM-1.3 Step Repetition (looping), FM-1.4 Loss of Conversation History (memory leaks), and FM-1.5 Unaware of Termination.
  • FC2: Inter-Agent Misalignment (The Communication): Runtime communication failures, including FM-2.2 Fail to Ask for Clarification and FM-2.3 Task Derailment.
  • FC3: Task Verification (The Quality Control): Output quality failures, such as FM-3.1 Premature Termination and FM-3.3 Incorrect Verification (hallucinating success).

Diagnostic Experiment: IT-Bench SRE Traces

Researchers annotated 310 SRE execution traces across three model classes to analyze their distinct failure signatures. The models showed vastly different performance levels based on mean recall:

  • Gemini-3-Flash: 75.5% Mean Recall
  • Kimi-K2: 28.6% Mean Recall
  • GPT-OSS-120B: 12.4% Mean Recall

Failure Density and Complexity

The study found a direct correlation between model strength and the "surgical" nature of its failures. Stronger models exhibit fewer, more isolated failure modes per trace, making them easier to debug:

  • Gemini-3-Flash: 2.6 failure modes per failed trace (Surgical failures).
  • Kimi-K2: 4.7 failure modes per failed trace.
  • GPT-OSS-120B: 5.3 failure modes per failed trace (Cascading collapse).

While Gemini-3-Flash typically fails due to a single isolated bottleneck, GPT-OSS-120B suffers from compounding errors where an early reasoning mismatch poisons the context and leads to total derailment.

Distinguishing Fatal vs. Non-Fatal Failures

MAST enables the classification of failures into two buckets based on whether the system can recover from them:

Recoverable / Structural Failures

These flaws appear frequently even in successful runs and are often necessary for the task. For example, FM-1.3 Step Repetition was present in over 90% of successful Kimi-K2 runs, as SRE tasks often require querying the same metric multiple times to verify stability.

Fatal / Decisive Failures

These errors strongly correlate with failure and are typically unrecoverable:

  • FM-3.3 Incorrect Verification: The strongest predictor of failure across all models. Gemini-3-Flash showed a 52% increase in this mode in failed traces compared to successful ones.
  • FM-1.5 Unaware of Termination Conditions and FM-2.6 Reasoning Action Mismatch are also primary drivers of failure.

Model-Specific Case Studies and Interventions

Gemini-3-Flash: Overconfidence

Gemini-3-Flash is efficient but tends to assume success without rigorous proof. Its failure signature is dominated by verification errors.

Recommended Fix: Implement an external verification gate. Require hard tool-based evidence (e.g., a healthy metric threshold) before allowing the agent to exit, rather than letting the LLM grade its own homework.

Kimi-K2: The Termination Crisis

Kimi-K2 struggles with recognizing when a task is complete. It exhibits a massive spike in Premature Termination (+46%) and Unaware of Termination Conditions (+43%). Additionally, FM-2.6 (Action-Reasoning Mismatch) is present in 92% of its failures, where the model identifies the correct step but executes an irrelevant command.

Recommended Fix: Use a deterministic state machine to enforce termination and loop control outside the model.

GPT-OSS-120B: Systemic Instability

This model shows a fundamental inability to maintain internal state. It loses conversation history (FM-1.4) in 24% of traces—compared to 0% for Gemini-3-Flash—causing it to "forget" the original alerts it was triaging.

Recommended Fix: Implement aggressive context hygiene and early error detection to prevent minor reasoning mismatches from compounding into total derailment.

Engineering Roadmap for Enterprise Agents

Based on the MAST analysis, the researchers suggest three primary architectural interventions to improve agent reliability:

  1. Externalize Verification: Require hard evidence before exit to mitigate overconfidence in frontier models.
  2. Externalize Loop Control: Move termination and loop detection outside the model using Finite State Machines to prevent infinite loops or premature exits.
  3. Force Clarification: Make ambiguity a first-class branch in the agent graph to prevent smaller models from assuming incorrect paths when inputs are ambiguous.

Sources