microsoft/OpenRCA

[ICLR'25] OpenRCA: Can Large Language Models Locate the Root Cause of Software Failures?

What it solves

OpenRCA tackles the challenge of measuring whether large language models (LLMs) can actually find the root cause of software failures. In real operations, this requires looking at huge amounts of telemetry data — like KPIs, dependency traces, and logs — and reasoning through complex system dependencies. Most benchmarks don't test this integrated skill, so OpenRCA provides a standard way to see how well models and agents perform at this task.

How it works

OpenRCA gives you a dataset with natural language queries and ground-truth root cause records. Each query asks the model to locate the failure time, the failing component, and the reason. The telemetry data is structured per system (Telecom, Bank, Market) with logs, metrics, and traces stored per date. You run an evaluation script that compares your model's predictions (as JSON-like strings) against the ground truth and produces a report. OpenRCA also includes a baseline agent called RCA-agent. Instead of feeding raw telemetry into the model (which would create very long contexts), RCA-agent uses Python code to retrieve and analyze data on its own, letting the model focus on reasoning. The agent runs in a secure sandbox — a fresh Docker container per diagnosis, no network, read-only filesystem, limited resources, and a 60-second deadline per step. Security tests are included, and you can reproduce the paper's results with your own API keys.

Who it's for

This is for researchers and engineers building or evaluating LLM-based systems for IT operations, site reliability, and automated debugging. It’s especially useful if you’re developing agents that must handle telemetry data or if you want to benchmark your own model or agent against a standard set of root cause analysis tasks. The reconstruction tool also lets you create new tasks from your own telemetry, so it can adapt to different software environments.

Highlights

  • Includes telemetry datasets from three different software systems: Telecom, Bank, and Market.
  • Provides an evaluation harness that accepts any prediction format with a “prediction” field, so you can submit results from custom agents.
  • Offers a secure, reproducible baseline agent (RCA-agent) with a hardened Docker sandbox that blocks network, restricts filesystem access, and enforces tight resource limits.
  • Includes security regression tests that verify sandbox isolation, secret protection, and cleanup (some tests require opt-in with a flag).
  • Supports regenerating random queries or building entirely new tasks from your own telemetry via a task specification JSON file.
  • Documents common pitfalls like timezone alignment (all faults are in UTC+8) and sampling-frequency mismatches, which help avoid false negatives.

Related

  • Project
  • Project
  • Project
  • Project
  • Project