microsoft/OpenRCA

What it solves

OpenRCA addresses the challenge of evaluating whether Large Language Models (LLMs) can perform root cause analysis (RCA) in complex software operating scenarios. It provides a standardized benchmark to test if models can analyze vast amounts of telemetry data to identify why a system failure occurred.

How it works

The project provides a dataset containing telemetry from different industries (Telecom, Bank, Market) consisting of KPI time series, dependency trace graphs, and semi-structured logs. To solve these tasks, LLMs must reason across these different data types and understand system dependencies.

As a baseline, the project introduces RCA-agent, which uses Python for data retrieval and analysis. This approach prevents the model from being overwhelmed by overly long contexts, allowing it to focus on reasoning and scaling to larger telemetry datasets.

Who it’s for

• AI researchers studying the reasoning capabilities of LLMs in technical diagnostics.
• Developers building AI agents for automated system monitoring and troubleshooting.
• Engineers interested in benchmarking root cause analysis tools.

Highlights

• Multi-modal Telemetry: Includes logs, metrics, and traces for comprehensive analysis.
• RCA-agent Baseline: A Python-based agent that retrieves and analyzes data to handle large-scale telemetry.
• Customizable Tasks: Tools to generate new tasks from existing or private telemetry data.
• Industry-specific Datasets: Includes scenarios from Telecom, Banking, and Market sectors.