shengshu-ai/minWM

A Minimal and Elegant Framework & Tutorial for Real-Time Interactive World Models

What it solves

minWM is a full-stack framework designed to help researchers and developers transition from a standard text-to-video (T2V) foundation model to a real-time, action-conditioned video world model. It removes the need for users to reverse-engineer complex pipelines by providing a complete, open-source path from data preparation to training and inference.

How it works

The framework implements a four-stage pipeline to transform bidirectional diffusion models into 4-step autoregressive (AR) student models for real-time performance:

  1. Data Preparation: Processes raw videos and camera trajectories (poses) into training-ready latents.
  2. Bidirectional SFT: Initial supervised fine-tuning of the foundation model.
  3. Distillation: A multi-stage process involving Teacher Forcing AR diffusion, Causal ODE/CD distillation, and Asymmetric DMD with self-rollout.
  4. Inference: Uses 4-step DMD inference with support for camera-trajectory control via pose strings (e.g., "a4,w8") or JSON files.

It supports multiple backbones, including Wan 2.1 and HunyuanVideo 1.5, using a unified trainer and dataset abstraction.

Who it’s for

It is primarily aimed at graduate students, independent researchers, and junior labs entering the world-model space who want a structured, runnable framework rather than a standalone model.

Highlights

  • Full-Stack Pipeline: Open-sources the entire data $\rightarrow$ training $\rightarrow$ inference flow.
  • Multi-Backbone Support: Compatible with both Wan 2.1 and HunyuanVideo 1.5 architectures.
  • LLM-Assisted Debugging: Includes "Claude Skills" to help users diagnose training failures (like loss NaN or camera drift) and integrate new backbones.
  • Real-Time Performance: Distills models down to 4-step inference for interactive video generation.

Related

  • Project
    simchowitzlabpublic/nano-world-model

    A minimalist framework for training video world models based on diffusion-forcing, used for future video prediction, 3D reconstruction, and robotic planning.

  • Project
    boundless-large-model/boundless-world-model

    BWM is a physically consistent, action-conditioned video world model that serves as a high-fidelity simulator for robotic manipulation tasks.

  • Project
    OpenWAM-Official/OpenWAM

    An open research stack for systematically developing World-Action Models (WAMs) that combine world knowledge and action learning for robotics.

  • Project
    open-gigaai/giga-world-1

    An open-source framework for training and deploying robot world models to evaluate robot policies through controllable video generation.

  • Project
    Ma-Zhuang/OmniNWM

    OmniNWM is a research codebase that builds a panoramic, multi‑modal world model for autonomous‑driving simulation. It jointly generates RGB, semantic, depth, and 3‑D occupancy maps from a vehicle trajectory, uses normalized Plücker ray‑maps for precise control, and provides occupancy‑based dense rewards for closed‑loop policy evaluation. The repo includes installation steps (including a manual patch to `transformers`), data preparation instructions for nuScenes, pretrained checkpoint download links, and scripts for inference, out‑of‑distribution testing, and staged training.