showlab/Show-Harness

Show‑Harness – A VLM‑driven Embodied Agent

What it is

• An open‑source harness that lets a vision‑language model (VLM) control a robot through a tiny, token‑based action vocabulary.
• Provides two operating modes:
  1. Zero‑shot – a large, closed‑source VLM (e.g., Gemini, GPT‑4‑V) is queried directly; the model decides each action token on‑the‑fly.
  2. Fine‑tuned – a smaller, open VLM is LoRA‑adapted on robot demonstration data and emits one action token per step.
• Works with multiple robot platforms (Franka, AgileX Piper, Isaac‑Lab, LIBERO) and simulators, all using the same prompt set and action vocabulary.

Key components

Component	Role
core/	Main loop that talks to the VLM, builds prompts, logs, and maps VLM tokens to the interpreter for the chosen robot.
interpreters/	Platform‑specific code that turns a token (e.g., GRIP_OPEN) into real robot commands (impedance control, joint streaming, etc.).
gumi/	A lightweight browser UI for collecting demonstrations. Users press keys (WASD, arrows) to drive the robot; each step is saved as an (observation, action) pair for training.
plugins/	Optional “plug‑ins” that can be toggled (perception, planning, recovery, etc.) without changing the core code.
train/	Scripts that convert the GUMI rollouts into a dataset, register it with 🤗 Datasets, and fine‑tune a VLM using LLaMA‑Factory (LoRA adapters).
scripts/	Helper utilities: environment setup, ROS/RealSense integration, robot calibration, serving a VLM with vLLM, and evaluation scripts for simulators.
configs/	Layered YAML configs – defaults, site‑specific robot settings, and optional overrides.

How to get started

1. Set up environments – run bash scripts/setup.sh base for the harness, … serve for a vLLM server, and optionally … libero <checkout> for a simulator.
2. Collect data – launch the GUMI UI (gumi/collect_rollouts_web.py) in simulation (--sim) or on real hardware; drive the robot with the keyboard and the system records ready‑to‑train pairs.
3. Run a robot – configure configs/site/*.yaml, add your API key in configs/secrets.env, run scripts/check_setup.py then scripts/run_real.py (zero‑shot) or scripts/run_real_mvtoken.py (fine‑tuned).
4. Fine‑tune – download one of the released LoRA adapters from 🤗 Hub, or train your own with the train/ pipeline.

Released resources

• Models: Five LoRA adapters for Qwen‑3.5 (0.8B, 2B, 4B, 9B) and Gemma‑4B, plus a simulation‑only adapter. Hosted at showlab/Show‑Harness‑VLMs.
• Dataset: Real rollouts from Franka/Piper and simulated data from RoboLab/ManiSkill, available at showlab/Show‑Harness‑Data.
• Paper & demo – arXiv pre‑print (2609.10522) and a short video showing the VLM playing a robot.

Why it matters

• Demonstrates that a single VLM interface can drive many robot bodies, removing the need for robot‑specific policy networks.
• The GUMI collector makes it easy to generate high‑quality, token‑level demonstrations without expensive tele‑operation rigs.
• The plugin architecture lets researchers add or remove perception/planning modules (e.g., multi‑view guidance, sub‑goal planning) with a single boolean flag, facilitating ablation studies.

Typical use‑cases

• Researchers wanting to test zero‑shot VLM control on a new robot platform.
• Teams that need a quick pipeline to collect robot demonstrations and fine‑tune a lightweight VLM.
• Developers building custom perception or planning plugins that plug into the same harness.

Citation

@article{chen2026show,
  title={Show‑harness: Just a VLM agent can play robots},
  author={Chen, Yanzhe and Bai, Zechen and Cao, Zhijun and Zeng, Wenzheng and Lin, Kevin Qinghong and Lin, Yiqi and Liang, Guoqiang and Ma, Kevin Yuchen and Huang, Qiming and Shou, Mike Zheng},
  journal={arXiv preprint arXiv:2609.10522},
  year={2026}
}

---

All details are taken directly from the repository’s README; no additional features are inferred.