robbyant-research/Zero-WAM

What it solves

Zero-WAM addresses the challenge of zero-shot cross-task robotic manipulation. It enables robots to execute tasks they were never trained on by using deployment-time context—such as human demonstration videos or language instructions—rather than requiring new robot-specific training data or parameter updates for every new task.

How it works

The project treats human demonstration videos as visual task specifications. It uses a causal video-action policy that predicts future robot observations and executable actions based on the provided context. To train this, the researchers developed the "HumanGen" pipeline, which uses VLMs, image editors, and video generation models to convert existing robot trajectories into semantically matched human manipulation videos, creating a large-scale dataset of human-robot in-context learning (ICL) pairs.

Who it’s for

This technology is designed for robotics researchers and engineers working on open-ended task generalization and embodied AI, specifically those aiming to reduce the reliance on massive amounts of robot-collected data for every new manipulation task.

Highlights

• In-Context World-Action Modeling: A single policy that supports both language and human videos as task instructions.
• HumanGen Pipeline: A scalable system that generated 74.2K human-robot ICL pairs across 8.6K tasks.
• Future Chunk Prediction: An objective designed to reduce shortcut learning and improve the model's reliance on human video prompts.
• Zero-Shot Performance: Demonstrated success in RoboTwin 2.0 and real-world unseen configurations without updating model parameters.