ShuangLI59/unified_video_action

What it solves

Unified Video Action (UVA) model addresses the challenge of training robotic manipulation policies that can both predict future video frames (visual foresight) gọi là visual foresight and execute precise actions. It aims to create a unified framework where video generation and action prediction are handled together, improving the robot's ability to generalize across different tasks.

How it works

UVA uses a two-stage training process to optimize performance:

1. Video Generation Stage: The model is first trained exclusively on a video generation task to learn visual representations of how the world changes.
2. Joint Video and Action Stage: The model is then fine-tuned to simultaneously predict both future video frames and the corresponding robotic actions.

It leverages a pretrained VAE and an image generation model (MAR) as a foundation. For real-world deployment, it supports various hardware setups, including the ARX X5 robot, and incorporates techniques like distinct history frequencies to handle differences between training data and real-time control frequencies.

Who it’s for

Robotics researchers and developers working on imitation learning, video-based action prediction, and embodied AI who want to integrate visual foresight into robotic control.

Highlights

• Two-Stage Training: Improves stability and performance by separating visual learning from action learning.
• Multi-Task Support: Demonstrated on simulation tasks (PushT, Libero10) and real-world tasks (cup arrangement, towel folding, mouse arrangement).
• Real-World Deployment: Includes specific instructions for ARX X5 robot and UMI hardware integration.
• Extensible Architecture: Provides clear paths for adding new tasks and custom models.