thu-ml/DiT-Extrapolation

What it solves

This project addresses the limitation of Diffusion Transformers (DiT) when generating content beyond their trained length or resolution. Specifically, it enables "extrapolation," allowing pre-trained models to generate longer videos (e.g., extending a 5s video to 11s) and higher-resolution images without requiring extensive retraining.

How it works

The core technique, RIFLEx, modifies the 1D Rotary Positional Embedding (RoPE) by reducing the intrinsic frequency of a specific component to ensure the generated content stays within a single period after extrapolation. This is achieved by adding a single line of code to the original RoPE implementation, making it a plug-and-play solution. The repository also includes implementations for UltraViCo and UltraImage for further breaking extrapolation limits in video and image generation.

Who it’s for

It is designed for researchers and developers working with video and image diffusion models who want to increase the output length or resolution of existing SOTA models like HunyuanVideo, CogVideoX, Wan2.1, Flux, and Qwen-Image.

Highlights

• Plug-and-Play: Can be applied to pre-trained models with minimal code changes.
• Temporal Extrapolation: Extends video length (e.g., HunyuanVideo from 5s to 11s, CogVideoX from 6s to 12s).
• Broad Model Support: Compatible with multiple SOTA architectures including HunyuanVideo, CogVideoX, Wan2.1, Flux, and Qwen-Image.
• Training-Free Option: Offers both training-free and fine-tuned versions for different performance needs.