robbyant-research/Zero-WAM

What it solves

Zero-WAM addresses the challenge of zero-shot cross-task robotic manipulation. It enables robots to perform tasks they were never trained on by using deployment-time context—specifically human demonstration videos or language instructions—as visual task specifications.

How it works

Zero-WAM treats robotic task generalization as an in-context world-action modeling problem. It uses a causal video-action policy that predicts future robot observations and executable actions based on the provided context.

To support this, the project introduces the HumanGen pipeline, which synthesizes semantically matched human video instructions from robot trajectories using a VLM, an image editor, and a video generation model. The model is trained with an in-context future chunk prediction objective to prevent shortcut learning and ensure the robot relies on the human video prompts for guidance.

Who it’s for

This project is for robotics researchers and developers working on general-purpose manipulation policies and zero-shot task generalization.

Highlights

• In-Context Learning for Robotics: Supports both human videos and language as task instructions for a single causal policy.
• HumanGen Pipeline: A scalable system that generated 74.2K human-robot ICL pairs across 8.6K tasks to bridge the gap between human demonstrations and robot execution.
• Zero-Shot Performance: Demonstrated success on unseen tasks in the RoboTwin 2.0 environment and real-world configurations without updating model parameters.
• Task-Diverse Training: Prioritizes task diversity over raw trajectory count, utilizing a mix of public robotic datasets and synthesized human data.