RoboDojo-Benchmark/RoboDojo

RoboDojo – A Unified Sim‑and‑Real Benchmark for Generalist Robot Manipulation

What it is – RoboDojo is an evaluation‑only benchmark that lets researchers measure how well a robot‑manipulation policy works both in simulation (Isaac Sim) and on real hardware. It ships a simulator client, a collection of 42 simulated tasks and 18 real‑world tasks across three robot platforms, and tooling to run those tasks automatically and aggregate scores on a public leaderboard.

Why it matters – Current robot‑learning benchmarks often focus on a single simulated scene or a narrow set of real‑world tasks, making it hard to tell whether a policy truly generalises. RoboDojo deliberately mixes:

• Five capability dimensions (generalisation, memory, precision, long‑horizon, open‑ended) to probe different skills.
• Heterogeneous parallel simulation that runs many tasks concurrently on Isaac Sim, giving fast feedback.
• Physically grounded assets (rigid, articulated, deformable objects) in a single, configuration‑driven scene.
• Real‑robot evaluation on three embodiments (Piper X, Piper, ARX X5) with the same task definitions used in simulation.

How it works

1. Install the repository (Python 3.11, Isaac Sim 5.1, Isaac Lab 2.3) and download the provided assets and data.
2. Pick a policy that lives in the companion project XPolicyLab (the policy must expose an eval.sh and a deploy.yml).
3. Run the helper script scripts/robodojo.sh (or the lower‑level scripts/eval_policy.sh) which launches the simulator, loads a task configuration from task/RoboDojo/, and communicates with the policy server.
4. After the run finishes, a one‑command summarize aggregates the per‑task scores into the standard leaderboard table.
5. Submit the results through the cloud evaluation pipeline to appear on the public leaderboard (both simulated and real‑world tracks are supported).

Key repository pieces

• env/ – Core simulator back‑end and managers that handle parallel task execution.
• env_cfg/ – YAML files describing simulator, scene, robot, and camera settings.
• task/RoboDojo/ – Task logic and per‑task configuration files.
• scripts/ – Entry‑points for evaluation (robodojo.sh, eval_policy.sh).
• XPolicyLab/ – Thin wrapper that loads a policy, starts a server, and talks back to RoboDojo.
• Assets/ – Downloaded robot models, objects, materials, and layout files.

Getting started – The documentation site (linked from the README) provides a Usage Overview with a step‑by‑step walkthrough, a Quick Evaluation guide to run a reference policy in minutes, and detailed pages for each simulated and real task.

Community & reproducibility – Scores are seed‑controlled and the entire evaluation stack is open source, so anyone can reproduce published results. A live leaderboard shows the current state‑of‑the‑art, and a community forum is linked from the project webpage.

Citation – If you use RoboDojo in a paper, cite the arXiv pre‑print (Chen et al., 2026) and also the companion XPolicyLab paper.

---

Bottom line – RoboDojo is a serious, open‑source benchmark for testing generalist robot manipulation policies across a wide variety of simulated and real tasks, with built‑in support for parallel evaluation and a public leaderboard.