IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST

Summary

IBM Research and UC Berkeley have introduced a diagnostic framework to move beyond simple success-rate metrics in evaluating agentic LLM systems for IT automation. By applying the Multi-Agent System Failure Taxonomy (MAST) to the ITBench benchmark, the researchers identified specific, structured failure modes in models performing SRE, Security, and FinOps tasks.

Key Points

• MAST Taxonomy Structure: The Multi-Agent System Failure Taxonomy (MAST) categorizes 14 distinct failure patterns into three categories: System Design Issues (FC1), Inter-Agent Misalignment (FC2), and Task Verification (FC3).
• Model Performance Variance: Testing across three model classes revealed significant disparities in Mean Recall: Gemini-3-Flash (75.5%), Kimi-K2 (28.6%), and GPT-OSS-120B (12.4%).
• Failure Density: Frontier models like Gemini-3-Flash exhibit "surgical" failures (2.6 failure modes per trace), whereas large open models like GPT-OSS-120B suffer from "cascading" failures (5.3 failure modes per trace) where early reasoning errors compound.
• Primary Failure Predictor: FM-3.3 (Incorrect Verification) was identified as the strongest predictor of failure across all tested models, where agents declare task success without verifying ground truth.
• Termination Instability: Kimi-K2 demonstrated specific instability in loop control, showing a 46% increase in Premature Termination and a 43% increase in being Unaware of Termination Conditions.

Technical Details

The research utilized 310 annotated ITBench SRE execution traces to analyze how agents interact with tools in Kubernetes and cloud environments. The study distinguishes between "Non-Fatal" (benign) failures—such as FM-1.3 (Step Repetition), which is common in troubleshooting—and "Fatal" failures that directly cause task collapse.

A critical technical finding is the behavior of FM-3.3 (Incorrect Verification), which caused a 52% increase in failed traces for Gemini-3-Flash. To mitigate these failures, the researchers suggest moving logic out of the LLM's reasoning loop. Recommended engineering interventions include:
- Externalized Verification: Implementing hard tool-based evidence requirements (e.g., checking a specific metric threshold) rather than allowing the LLM to self-grade.
- Deterministic Loop Control: Implementing Finite State Machines (FSMs) or explicit loop detectors outside the model to manage termination and prevent infinite tool-calling loops (FM-1.5).
- Ambiguity Handling: Implementing "clarify-or-read-only" branches in the agent graph to address FM-2.2 (Failure to Ask for Clarification), particularly for smaller or less capable models.

Impact / Why It Matters

This research provides a blueprint for developers to move from "black-box" evaluation to structured error mitigation in enterprise agent workflows. By identifying specific failure vectors, engineers can implement targeted architectural safeguards—like external verification gates and explicit termination logic—to build more reliable autonomous systems.