showlab/Show-Harness

What it solves

Show-Harness addresses the difficulty of controlling diverse robotic hardware using Vision-Language Models (VLMs). It provides a unified semantic interface that allows a VLM to "play" a robot by reasoning over discrete action units, removing the need for robot-specific training for every new piece of hardware. This allows both high-end frontier models to operate robots zero-shot and smaller open-source models to be fine-tuned into capable policies with minimal compute resources.

How it works

The system uses an "embodied harness" that acts as a compact semantic interface between the VLM and the robot. The VLM emits action tokens, and embodiment-specific interpreters ground these tokens into deterministic physical motions.

Key components include:

• Two Operating Modes: A zero-shot mode for frontier VLMs and a fine-tuned mode for smaller VLMs that emit one action token per step.
• GUMI: A browser-based teleoperation tool that allows humans or GUI agents to demonstrate tasks by mapping action units to keys/buttons, recording these as training-ready (observation, action) pairs.
• Interpreters: Hardware-specific layers for robots like Franka, AgileX Piper, and simulators (ManiSkill, Isaac Lab) that share a common vocabulary.
• Plugins: Modular components that can be toggled to add capabilities like multi-view guidance, proprioception, subtask planning, and failure recovery.

Who it’s for

It is designed for robotics researchers and developers who want to use VLMs as robot controllers across different hardware platforms without needing to build custom control pipelines for every robot.

Highlights

• Embodiment-agnostic: Works across Franka, AgileX Piper, and various simulators using a single vocabulary and prompt set.
• Efficient Fine-tuning: Small VLMs can become capable policies with less than a few H200 GPU-hours of training.
• GUMI Teleoperation: Enables data collection via a web browser without requiring specialized teleoperation hardware.
• Modular Plugin System: Features a suite of "ablation-grade" plugins for perception, reasoning, and action recovery that can be enabled or disabled via a simple boolean.