NVlabs/rcm

What it solves

rCM addresses the quality and diversity bottlenecks found in few-step video generation. Specifically, it overcomes the limitations of existing continuous-time consistency distillation methods (like sCM) by combining forward-divergence (offline) and reverse-KL (on-policy) distillation frameworks to maintain high visual quality while preserving the diversity of the original teacher model.

How it works

The project implements a score-regularized continuous-time consistency model. It uses a joint distillation framework that complements Consistency Models (CM) and Diffusion Model Distillation (DMD) to enable high-quality video generation in just 2 to 4 sampling steps.

For streaming video and interactive world models, it introduces Causal-rCM, which integrates teacher-forcing (forward-divergence) and self-forcing (reverse-divergence) training stacks to support autoregressive video diffusion distillation.

Who it’s for

AI researchers and engineers working on video diffusion models, specifically those looking to scale distillation to large models (10B+ parameters) or develop streaming video generation and interactive world models.

Highlights

• Few-Step Generation: Produces high-quality, diverse videos in 1–4 steps.
• Scalable Infrastructure: Supports scaling to 10B+ parameter models using FSDP2, Ulysses Context Parallel (CP), and Selective Activation Checkpointing (SAC).
• Custom Kernels: Includes an open-sourced FlashAttention-2 Jacobian-vector product (JVP) kernel.
• Causal Extension: Provides a unified recipe for autoregressive training, including optimized causal inference with quantized KV cache and length extrapolation.