IBM/AssetOpsBench

AssetOpsBench – AI Agents for Industrial Asset Operations & Maintenance

What it is – A research‑grade, open‑source framework that lets you build, orchestrate, and evaluate large‑language‑model (LLM) agents that work with real‑world industrial data (sensor streams, work‑order systems, failure‑mode catalogs, etc.). It bundles:

• Domain‑specific MCP servers – ready‑made tool APIs for IoT assets, failure‑mode knowledge bases, time‑series foundation models, work‑order management, and vibration analysis.
• Agent back‑ends – plug‑and‑play orchestrators (Plan‑Execute, Deep Agent, Claude‑based, OpenAI‑based) that follow the ReAct pattern and can call the MCP tools as “actions”.
• Benchmark suite – > 140 reproducible scenarios covering 9 asset classes, each with a defined multi‑step workflow and an evaluation pipeline that scores agents on reasoning, execution, and data‑handling dimensions.
• Evaluation infrastructure – leaderboards, LLM‑based judges, and multi‑dimensional metrics (reasoning quality, tool‑call correctness, latency, etc.).

---

Why it matters

Industrial “Industry 4.0” sites generate massive streams of sensor data and maintain complex work‑order systems. Turning that data into autonomous decision‑making (e.g., detecting an anomaly, generating a work order, selecting a forecasting model) requires agents that can:

1. Understand natural‑language requests from engineers.
2. Retrieve and manipulate structured data via domain‑specific tools.
3. Plan and execute multi‑step workflows reliably. AssetOpsBench provides the whole stack so researchers can focus on the LLM reasoning part rather than building the plumbing from scratch.

---

Core components

Component	What it provides	Typical use
MCP (Model Context Protocol) servers	REST‑style tool APIs for IoT, Failure‑Mode‑SR (FMSR), Time‑Series FM (TSFM), Work‑Order (WO), Vibration, plus shared utilities.	Agents call list_assets, generate_workorder, run_recipe, etc., to interact with live‑like industrial data.
Agent frameworks	• plan_execute – simple plan‑and‑execute loop for any LLM. • deep_agent – hierarchical planning with sub‑agents and a virtual filesystem. • claude_agent / openai_agent – ReAct‑style orchestrators that delegate tool calls to Claude or OpenAI models.	Choose the level of sophistication you need; all share a common MCP client library.
Benchmark scenarios	141+ curated tasks (e.g., “List all sensors of Chiller 6”, “Generate a work order for an anomaly”).	Run python -m assetopsbench.run --scenario "…" or use the Hugging‑Face playground to see an agent in action.
Evaluation pipeline	Runs agents, records tool calls, scores with LLM‑judge (Llama‑4‑Maverick‑17B) across six criteria (reasoning, tool use, latency, etc.).	Produces leaderboard tables comparable across models.
Data & playground	Dataset on Hugging‑Face, interactive Space, Colab notebook, Kaggle benchmark page.	Easy for newcomers to experiment without installing anything locally.

---

Getting started (quick)

# Clone and install the package (editable mode)
git clone https://github.com/IBM/AssetOpsBench.git
cd AssetOpsBench
pip install -e .

# Run a simple demo scenario (once the MCP servers are up)
python -m assetopsbench.run --scenario "List all sensors of Chiller 6 in MAIN site"

For a zero‑install experience, open the Colab notebook linked in the README or the HF Playground to watch an LLM agent query the IoT server and produce a response.

---

Who should use it

Audience	What they gain
Researchers in LLM agents, tool‑use, or industrial AI – a ready‑made testbed with realistic data and a public leaderboard.
Industrial AI engineers – a reference implementation of how to expose plant data as tool APIs and connect them to LLM planners.
Students – hands‑on tutorials (MCP tutorial at KDD 2026, AAAI 2026 lab) and a Colab notebook for coursework on agentic AI.
Competition organizers – the same benchmark powers the IJCAI 2026 live challenge and CODS 2025 competition.

---

License & community

• License: Apache 2.0 (permissive, commercial‑friendly).
  * Contributions: Open to PRs – the README invites universities to add extensions.
  * Support channels: GitHub issues, the INSTRUCTIONS.md for setup, and a set of tutorial PDFs linked in the repo.
  * Publications: Over a dozen papers across KDD, ACL, ICLR, AAAI, EMNLP, etc., providing deep technical background and evaluation results.

---

TL;DR

AssetOpsBench is an end‑to‑end, open‑source benchmark that lets you prototype and compare LLM‑driven agents for real‑world industrial maintenance tasks. It ships with domain‑specific tool servers (MCP), multiple agent orchestrators, a large library of realistic scenarios, and an evaluation framework that has already been used in major AI conferences and competitions.