allenai/vla-evaluation-harness

One framework to evaluate any VLA model on any robot simulation benchmark.

What it solves

Evaluating Vision-Language-Action (VLA) models is typically fragmented because every robot simulation benchmark (like LIBERO or CALVIN) has its own dependencies, observation formats, and evaluation protocols. This often leads research teams to maintain separate, private forks for each benchmark, resulting in inconsistent results and a lack of standardized testing under real-time conditions.

How it works

The harness provides an abstraction layer that decouples models from benchmarks through two primary mechanisms:

  • Dockerized Benchmarks: Each benchmark environment is packaged as a standalone Docker image to ensure exact reproducibility and avoid dependency conflicts.
  • Standalone Model Servers: Models are served as standalone scripts (using uv) with inline dependency declarations, allowing them to be integrated once and then evaluated across any supported benchmark.

To speed up evaluation, the system uses Episode Sharding (splitting tasks across multiple processes) and Batch Inference (grouping requests into a single GPU forward pass), which can increase throughput by up to 47x.

Who it’s for

Researchers and developers working on VLA models and embodied AI who need a unified, reproducible way to evaluate their models across multiple robot simulation benchmarks.

Highlights

  • Massive Integration Matrix: Supports 18 benchmarks and 13 official model servers, including OpenVLA, $\pi_0$, and GR00T.
  • High Throughput: Batch parallel evaluation can reduce wall-clock time for 2,000 episodes from 14 hours to 18 minutes on a single H100.
  • Zero Setup: Uses Docker and uv scripts to eliminate manual environment configuration.
  • Unified Leaderboard: Aggregates data from over 2,000 models across 18 benchmarks.
  • Observability: Built-in SQLite recording for episode results and optional video capture for visual debugging.

Related

  • Project
    OpenBMB/DeepThinkVLA

    DeepThinkVLA is a Vision‑Language‑Action model that inserts an explicit chain‑of‑thought reasoning step before generating robot actions. Using a hybrid decoder (autoregressive reasoning → bidirectional parallel action generation), the authors train on a newly curated embodied CoT dataset and then refine with outcome‑driven reinforcement learning. The model achieves 97 % average success on the LIBERO benchmark, runs at only 0.175× the latency of a pure autoregressive baseline, and transfers zero‑shot to the newer LIBERO Plus suite.

  • Project
    sou350121/VLA-Handbook

    A continuously‑updated, open‑source handbook that curates VLA (vision‑language‑action) research, engineering tricks, community notes, and industry intel, providing reproducible scripts, hardware guides, RSS feeds, and an AI‑assistant skill—all under a CC‑BY‑4.0 license.

  • Project
    dexmal/dexbotic

    Dexbotic is an open‑source PyTorch toolbox for Vision‑Language‑Action (VLA) research. It bundles pre‑trained VLA models (π0, CogACT, OFT, MemVLA, etc.), modular training/evaluation scripts, LoRA fine‑tuning, cloud‑ready distributed training, and deployment helpers for robots like UR5, Franka, and Unitree G1 SONIC. The project provides Docker images, extensive documentation, benchmark tables, and a unified realtime inference API, all under an MIT license.

  • Project
    dsta022/Loop-Engineering-for-VLA

    Loop Engineering for VLAnything is an end‑to‑end Python toolkit that lets you record, merge, audit, enrich, and iteratively improve multimodal robot‑learning datasets (RGB, RGB‑D, and future visuo‑tactile) for vision‑language‑action models. It provides a conservative, side‑car‑first quality audit, plug‑in semantic evaluators, language‑annotation pipelines, and a closed‑loop feedback loop for policy improvement, all compatible with the LeRobot data format and publishable to Hugging Face.

  • Project
    isaac-sim/IsaacLab-Arena

    Isaac Lab‑Arena is an alpha‑stage, open‑source extension to NVIDIA Isaac Lab that lets you build robot simulation environments by mixing reusable *scene*, *embodiment*, and *task* primitives. It assembles these at runtime, supports sequential task chaining, natural‑language object placement, and large‑scale parallel evaluation, making it easier to benchmark or train generalist robot policies. Installation works via the `uv` Python manager or a provided Docker image; the library is Linux‑only and requires an NVIDIA GPU and Isaac Sim. The project is under Apache 2.0 (with proprietary Isaac Sim components) and is intended for research rather than production.