Inspect Robots – Evaluation Framework for Physical AI

What it is

• An open‑source Python library that lets you define a robotics benchmark once and then run any robot‑control policy (LLM‑driven agents, vision‑language models, scripted scripts, etc.) on any compatible embodiment – a real arm, a simulated robot, or a mock world.
• It records everything: sensor streams, policy transcripts, configuration, and scores, and stores them as replayable logs.
• Live visualisation is provided via the Rerun viewer, which can stream camera images, joint trajectories and scores while the robot moves.

Key concepts

• Task – a description of what the robot should do (e.g. “place the fork on the plate”).
• Embodiment – the robot or simulator that executes actions. Embodiments are discovered via Python entry‑points; installing a plugin (e.g. inspect-robots-yam) makes a new robot appear.
• Policy – the decision‑making component. Policies can be:
  • a frontier LLM (via inspect-robots-agent),
  • a VLA served by XPolicyLab (xpolicylab),
  • a code‑as‑policy agent (capx),
  • a simple scripted policy, or
  • any custom client that follows the action contract.
• EvalLog – an immutable JSON record containing the full config, git revision, metrics, and (optionally) camera frames.
• Rerun – optional visualisation that records a .rrd file and can be streamed to a remote viewer.

Typical workflow

1. Install core + optional extras

   uv venv && uv pip install "inspect-robots[rerun]"
   # or just `inspect-robots` for the numpy‑only core
   

2. Add a robot plugin (example for the YAM bimanual arm)

   uv pip install inspect-robots-yam
   inspect-robots setup   # wizard creates ~/.config/inspect-robots/config.ini
   

3. Start any required policy server (e.g. MolmoAct2 for the molmoact2 policy).
4. Run a task

   inspect-robots "place the fork on the plate"
   

   • The command launches the Rerun viewer, streams data, and writes a log file.
5. Inspect results

   inspect-robots view logs/          # static HTML index
   inspect-robots inspect LOG.json   # pretty‑print a single log
   inspect-robots video LOG.json     # render stored frames to MP4
   

6. Iterate – you can add voice feedback, summarize failures, or retry with learned notes using the built‑in summarize command.

Supported robots & simulators (via first‑party plugins)

Robot / World	Embodiment name	Package	Action space
I2RT YAM bimanual arm	yam_arms	inspect-robots-yam	14‑D joint positions
Franka FR3 / Panda	franka	inspect-robots-franka	8‑D joint positions
AgiBot A2 Ultra	a2_arms	inspect-robots-agibot-a2	16‑D joint positions
Unitree G1	g1_arms	inspect-robots-unitree-g1	16‑D joint positions
SO‑ARM followers	so_arm	inspect-robots-so101	6‑D joint positions
WidowX 250S	widowx	inspect-robots-widowx	7‑D end‑effector pose
Any ROS‑compatible arm	ros	inspect-robots-ros	configurable
Isaac Lab simulation	isaacsim	inspect-robots-isaacsim	joint positions
CubePick mock world (no hardware)	cubepick	built‑in	2‑D delta pose

Extensibility

• New robots are added by publishing a Python package that registers an embodiment entry‑point and (optionally) a matching policy client.
• New policies are added similarly; the framework validates compatibility (action/observation shapes, control rate, etc.) before any motion starts.
• Optional extras (rerun, inspect-robots-agent, inspect-robots-capx, …) are installed separately, keeping the core lightweight (NumPy only).

Why it matters

• Provides a single, reproducible evaluation pipeline for real‑world robotics research, bridging the gap between simulation benchmarks and physical experiments.
• Guarantees auditability: every run is a self‑describing log that can be re‑scored offline.
• Supports frontier AI policies (LLM agents, code‑as‑policy) alongside classic VLAs, making it easy to compare emerging models on the same hardware.
• Built‑in safety guardrails (error taxonomy, action bounds, human‑in‑the‑loop reset) prevent unattended robots from running away.

Getting started resources

• Quickstart guide: docs/guide/quickstart.md
• Rig reference (how to write a robot plugin): docs/guide/rig-reference.md
• Concepts overview: docs/guide/concepts.md
• Full CLI reference: docs/guide/cli.md

---

Inspect Robots is currently in alpha; the API may change, so pin a released version if you depend on it.