IBM/AssetOpsBench

AssetOpsBench – AI Agents for Industrial Asset Operations & Maintenance

What it is – AssetOpsBench is an open‑source framework that lets researchers and engineers build, orchestrate, and evaluate large‑language‑model (LLM) agents that work with real‑world industrial data. It bundles:

• Domain‑specific “MCP” servers that expose common Industry 4.0 services (IoT sensor catalogs, failure‑mode libraries, time‑series model catalogs, work‑order management, vibration analysis, plus shared utilities).
• Agent back‑ends – ready‑made ReAct‑style orchestrators for OpenAI, Anthropic Claude, a generic “plan‑execute” runner, and a deeper “Deep Agent” that can spawn sub‑agents and use a virtual file‑system.
• A library of reproducible scenarios (141+ across 9 asset classes) that describe multi‑step maintenance or monitoring tasks such as “list all sensors for Chiller 6”, “generate a work order for an anomaly”, or “find a forecasting model and load it”.
• Evaluation pipelines that run agents on the scenarios, record their tool calls, and score the results with a dedicated LLM judge (Llama‑4‑Maverick‑17B) across six quantitative criteria.

Why it matters – Industrial plants generate massive streams of sensor data, maintenance logs, and equipment metadata. Turning that data into actionable decisions traditionally requires hand‑crafted pipelines. AssetOpsBench provides a standardised benchmark for the emerging class of agentic AI systems that can reason, call tools, and execute workflows autonomously. By exposing the same MCP APIs to every participant, the benchmark makes it possible to compare LLMs, prompting strategies, and planning algorithms on a level playing field.

Key components

Component	What it does
MCP (Model Context Protocol) servers	REST‑like services for IoT, Failure‑Mode‑Specification‑Repository (FMSR), Time‑Series‑Foundation‑Model (TSFM), Work‑Order (WO), Vibration analysis, plus shared utilities.
Agent frameworks	• plan_execute – simple plan‑and‑execute loop for any LLM. • deep_agent – hierarchical planning with sub‑agents and a virtual FS. • claude_agent / openai_agent – ReAct‑style orchestrators that treat the LLM as a tool‑calling agent.
Scenario library	141+ scripted tasks covering sensor queries, failure‑mode look‑ups, model discovery, work‑order creation, and multi‑domain end‑to‑end workflows.
Evaluation suite	Runs agents, logs tool usage, and scores trajectories with an LLM judge on reasoning, execution correctness, data handling, etc.
Datasets & playgrounds	Hugging‑Face dataset (ibm-research/AssetOpsBench), an interactive Space demo, a Kaggle benchmark page, and a Colab notebook that runs a sample agent without local install.

Typical workflow

1. Start the MCP servers (or connect to the hosted ones). They provide the “tools” the agent can call.
2. Pick a scenario from the library (e.g., List all assets in MAIN site).
3. Run an agent – e.g., python -m assetopsbench.run --scenario "..." – which will:
   • Prompt the LLM to plan.
   • Translate each step into an MCP tool call.
   • Collect results and feed them back to the LLM.
4. Evaluate – the built‑in evaluator compares the agent’s trajectory to a gold‑standard and produces a multi‑dimensional score.

Getting started

# Clone and install (editable mode)
git clone https://github.com/IBM/AssetOpsBench.git
cd AssetOpsBench
pip install -e .

# Quick demo on Colab (no local install needed)
# https://colab.research.google.com/github/IBM/AssetOpsBench/.../LLM_Agent.ipynb

For a full walkthrough see INSTRUCTIONS.md, which explains how to launch the MCP servers, configure the AOB_READONLY flag, and run the plan‑execute runner.

Ecosystem & extensions

• Datasets – the benchmark data are hosted on Hugging‑Face and Kaggle.
• Competitions – live challenges at IJCAI 2026 and CODS 2025 have used AssetOpsBench as the evaluation backend, attracting >500 agent submissions.
• University projects – dozens of research groups have forked the repo to add new asset classes, optimise the TSFM server, or experiment with confidence‑gated multi‑agent planning.
• Related code – IBM’s SPIRAL planning library, ReActXen IoT agent, and FailureSensorIQ QA dataset are linked from the README and can be used alongside AssetOpsBench.

Maturity – The core framework is actively maintained on the main branch. Earlier publication‑specific branches (IndustryAssetEQA, main-0.x) are kept for reproducibility. The project has been accepted to multiple top venues (KDD 2026, ACL 2026, ICLR 2026, etc.) and is already powering public AI competitions.

License – Apache 2.0 (per the badge in the README).

---

Bottom line – AssetOpsBench is a genuine, research‑grade benchmark that lets anyone experiment with LLM‑driven agents in realistic industrial maintenance settings, providing standardized tools, data, and evaluation metrics to accelerate the field of AI agents for Industry 4.0.