allenai/vla-evaluation-harness

vla‑evaluation‑harness – One framework to evaluate any VLA model on any robot‑simulation benchmark

What it is – A Python library (installable via pip install vla-eval) that lets you run a VLA (Vision‑Language‑Action) model against many robot‑simulation benchmarks (LIBERO, CALVIN, SimplerEnv, etc.) without manually wiring each benchmark’s dependencies. Benchmarks run inside Docker containers, while model code is served as a lightweight HTTP server (a model server). The harness coordinates the two, records results, and can do massive batch‑parallel runs.

Key features

• Huge matrix – 18 benchmarks × 2 456 reproduced models (as of the 2026 leaderboard). Official model servers for OpenVLA, π₀, GR00T, OFT, X‑VLA, MolmoBot, etc., plus community servers from dexbotic, starVLA, and 🤗 LeRobot.
• Zero‑setup Docker – Benchmarks are pre‑built Docker images; the harness pulls them automatically. No dependency conflicts between benchmarks.
• Batch‑parallel evaluation – Episodes are sharded across many processes and model inference is batched, giving up to ~47× speed‑up (2 000 LIBERO episodes in 18 min on a single H100).
• Python API – vla_eval.evaluate(model_server, config_path) lets you evaluate while training, as shown in the examples/pusht_train_eval/ script.
• Observability – Results saved as JSON or SQLite recordings; integration with Weights & Biases, TrackIO, and a built‑in export tool.
• Extensible – Skills (skills/add-benchmark/, skills/add-model-server/) and AI‑assisted templates help you add new benchmarks or model servers.

Typical workflow

1. Serve a model (GPU host):

   vla-eval serve --config configs/model_servers/openvla/libero.yaml
   

2. Run a benchmark (Docker by default):

   vla-eval run --config configs/benchmarks/libero/spatial.yaml
   

   Results appear under results/.
3. Optional batch run – launch many shards with ./scripts/run_sharded.sh -c <config> -n <num_shards> and enable max_batch_size in the model‑server config.

Installation – pip install vla-eval (requires Python 3.8+, SQLite 3.24+). For development, clone the repo and use uv sync.

Who should use it – Researchers building VLA agents who want reproducible, comparable evaluations across many robot simulation suites, and engineers needing fast, scalable benchmarking pipelines.

Where to learn more – README sections on Motivation, Batch Parallel Evaluation, the architecture doc, and the live leaderboard.