thu-ml/Causal-Forcing

What it solves

Causal Forcing addresses the challenge of creating high-quality, real-time interactive video generation models. It specifically targets the issue of exposure bias and motion dynamics degradation in autoregressive (AR) diffusion models, allowing for faster inference (1-4 steps) without sacrificing visual quality or motion fluidity.

How it works

The project implements a three-stage training pipeline to distill a complex diffusion model into a fast, autoregressive generator:

1. AR Diffusion Training: Establishes the base autoregressive diffusion model.
2. Causal Initialization: Uses either Causal ODE (which requires curated ODE-paired data) or Causal Consistency Distillation (Causal Forcing++, which uses ground-truth data to eliminate the need for ODE data curation) to create a theoretically correct initialization.
3. Asymmetric DMD: Applies Distribution Matching Distillation to further reduce the number of steps required for generation.

It supports both chunk-wise and frame-wise models, with the latter natively enabling Image-to-Video (I2V) generation by setting the first latent frame as the conditional image.

Who it’s for

This is for AI researchers and developers working on video generation, specifically those looking to implement real-time interactive video tools or long-form video generation (via extensions like Rolling Forcing).

Highlights

• Ultra-Low Latency: Offers 1-step and 2-step frame-wise models for extremely fast generation.
• High Fidelity: Outperforms previous methods like Self Forcing in both visual quality and motion dynamics.
• Versatile Modalities: Supports both Text-to-Video (T2V) and Image-to-Video (I2V).
• Long Video Support: Compatible with techniques like Rolling Forcing to generate minute-level videos.