yuantianyuan01/FastWAM

What it solves

FastWAM addresses the computational overhead and latency associated with World Action Models (WAMs) in robotics. Specifically, it explores whether "test-time future imagination" (predicting future video frames before predicting actions) is necessary for high performance, providing a way to predict actions directly from current observations to achieve significantly faster inference.

How it works

FastWAM utilizes a model architecture based on a denoising core (ActionDiT) and VAE encoding. It offers two primary modes of operation:

• First-frame mode (Fast-WAM): Predicts actions directly from the current observation, skipping the future video imagination step to reduce latency.
• IDM mode: Follows the traditional approach of imagining future video frames first and then predicting actions based on those imaginations.

To optimize performance, the project implements a compiled denoising core, batched VAE encoding, and a lightweight CUDA Graph backend, which together reduce inference time and training duration.

Who it’s for

This project is designed for robotics researchers and developers working on embodied intelligence, specifically those focusing on action prediction, world models, and improving the real-time performance of robot control policies.

Highlights

• Accelerated Inference: Achieves approximately 2x faster end-to-end inference compared to previous versions.
• Dual-Mode Capability: The "Optional IDM" variant allows a single model to switch between future imagination (IDM) and direct action prediction (First-frame) without retraining.
• Dataset Support: Native support for LeRobot 2.1 and 3.0 datasets, improving scalability for large-scale data.
• High Success Rates: Demonstrates strong performance on the LIBERO benchmark across spatial, goal, object, and long-horizon tasks.
• Optimized Pipeline: Includes precomputed T5 text embeddings and compiled training paths to increase throughput.