Junchao-cs/SolarWM

What it solves

SolarWM provides a comprehensive, open-source foundation for creating interactive video world models. It addresses the difficulty of training models that can generate long-horizon, camera-controlled video rollouts (spanning minutes to hours) while using only short training clips (around 5 seconds), all while remaining compatible with various existing video model backbones.

How it works

SolarWM uses a unified data infrastructure to process millions of video clips into a standardized format including camera geometry and captions. It then employs a three-stage training recipe to transform bidirectional video models into causal, autoregressive models:

1. Stage 0.5 (Bidirectional Adaptation): Establishes base representations for video, text, and camera conditioning.
2. Stage 1 (Teacher-Forced AnyFlow): Combines teacher forcing with AnyFlow loss to learn denoising and finite-step flow maps, removing the need for separate ODE initialization.
3. Stage 2 (Distribution Matching Distillation): Uses self-gradient forcing (SGF) to train a causal student model on its own rollouts with a frozen teacher and a trainable critic.

Who it’s for

Researchers and developers working on video generation, world models, and interactive AI environments who want a scalable framework to build upon diverse video backbones like Wan2.2, LTX-2.5, and MiniMax-H3.

Highlights

• Backbone Agnostic: Supports multiple 5B–33B model families across different architectures.
• Long-Horizon Inference: Enables real-time interaction and rollouts lasting minutes to hours without requiring long-sequence fine-tuning.
• Open Data Pipeline: Includes a reconfigurable infrastructure for 1.43 million canonical clips from 14 datasets.
• Simplified Training: A streamlined three-stage recipe that avoids complex ODE or consistency-distillation initialization.