liberomax/LIBERO-MAX

What it solves

LIBERO-MAX addresses the challenge of evaluating whether robot control policies can successfully adapt to unexpected environmental changes that occur while a task is already in progress. Many existing benchmarks test robots in static environments, but this project provides a framework to measure how performance degrades when the world shifts mid-execution.

How it works

The benchmark uses a "paired protocol" to isolate the impact of environmental changes. It compares a Base rollout (where the environment remains stable) against a Dynamic rollout (where the same task is performed, but a specific event is triggered mid-task). By comparing these two scenarios, researchers can measure exactly how much a specific change—such as a moved object, a camera shift, or new obstacles—causes a robot to fail.

Who it’s for

This project is designed for robotics researchers and developers working on robot policy evaluation. It provides a standardized way to test the robustness of AI-driven robot agents against real-world unpredictability.

Highlights

• 8,000 Paired Scenarios: Includes a comprehensive set of test cases covering geometry, observation, appearance, and path constraints.
• Lite Version: Offers a smaller 800-pair subset for rapid testing and development before running the full benchmark.
• Actionable Diagnostics: Identifies specific vulnerabilities, such as target relocation, which consistently causes the highest failure rates across tested policies.
• Performance Gap Measurement: Quantifies the success rate drop (typically 11% to 25.7%) that occurs when robots face mid-task environmental changes.