roboterax/video-prediction-policy-2

What it solves

VPP2 addresses the need for robots to better understand and interact with their environments by combining video prediction with action generation. It allows robots to "predict" future visual states before taking actions, which improves performance in complex tasks that require visual planning and memory.

How it works

The system uses a three-stage pipeline:

1. Event-level video pretraining to learn visual dynamics.
2. Visual planning through post-training and distillation to refine how the model anticipates outcomes.
3. Action learning using a mixture-of-transformers (MoT) architecture that translates visual predictions into specific robot movements.

For tasks requiring memory, the model can incorporate history frames into its input, allowing it to make decisions based on past events.

Who it’s for

This project is designed for researchers and developers working on robot learning, visual planning, and autonomous agents. It is particularly useful for those working with benchmarks like RoboDojo and LIBERO who need a framework to train and evaluate policies that bridge the gap between visual prediction and physical action.

Highlights

• State-of-the-art performance on the RoboDojo-Sim leaderboard.
• Supports zero-shot video prediction from a single image and text instruction.
• Includes pre-trained model weights for both video generation and policy evaluation.
• Provides modular training and evaluation workflows for RoboDojo and LIBERO benchmarks.