shengshu-ai/minWM

What it solves

minWM provides a complete, open-source pipeline for creating action-conditioned video world models. It simplifies the complex process of converting a standard bidirectional text-to-video (T2V) foundation model into a real-time, interactive world model that can respond to specific actions or camera trajectories.

How it works

The framework implements a multi-stage distillation process to transform foundation models into efficient, few-step autoregressive (AR) students. The pipeline consists of two main phases:

1. Phase 1: Bidirectional SFT - Initial supervised fine-tuning.
2. Phase 2: Distillation - A three-stage process involving Teacher Forcing AR Diffusion, Causal ODE/CD (Causal Forcing/Causal Forcing++), and Asymmetric DMD with Self Rollout to achieve 4-step real-time inference.

It supports multiple backbones, specifically Wan 2.1 and HunyuanVideo 1.5, and allows for condition injection (such as camera poses) using methods like ProPE.

Who it’s for

It is designed for graduate students, independent researchers, and junior labs who want to build video world models without having to reverse-engineer existing repositories from scratch.

Highlights

• Full-Stack Pipeline: Covers everything from data construction and processing to training and inference.
• Multi-Backbone Support: Works with both Wan 2.1 and HunyuanVideo 1.5 architectures.
• Real-Time Inference: Distills models down to a 4-step DMD inference process for fast generation.
• LLM-Assisted Onboarding: Includes "Claude skills" to help users debug training failures and integrate new model backbones.
• Camera Control: Supports precise camera-trajectory control via pose strings or JSON files.