thu-ml/Causal-Forcing

What it solves

Causal Forcing addresses the challenge of creating high-quality, real-time interactive video generation. It specifically aims to improve visual quality and motion dynamics in autoregressive (AR) diffusion models compared to existing methods like Self Forcing, while maintaining the same training budget and inference efficiency.

How it works

The project implements a multi-stage training pipeline to create efficient video generators:

1. AR Diffusion Training: The base stage for both chunk-wise and frame-wise models.
2. Initialization: The model is initialized using either Causal ODE (requiring ODE-paired data curation) or Causal Consistency Distillation (introduced in Causal Forcing++, which uses only ground-truth data and eliminates the need for ODE data curation).
3. Asymmetric DMD: A final distillation stage that enables the model to generate high-quality video in very few steps (e.g., 1, 2, or 4 steps).

Frame-wise models natively unify text-to-video (T2V) and image-to-video (I2V) generation by treating the first latent frame as the conditional image.

Who it’s for

This is for researchers and developers working on video world models, real-time interactive video generation, and efficient diffusion distillation techniques.

Highlights

• High Efficiency: Supports 1-step and 2-step frame-wise models for extremely low latency.
• Versatile Generation: Natively supports both T2V and I2V in frame-wise settings.
• Long Video Support: Compatible with extensions like Rolling Forcing to enable minute-level long video generation.
• Improved Dynamics: Outperforms Self Forcing in visual quality and motion dynamics.