State of Open Video Generation Models in Diffusers

Hugging Face has detailed the current landscape of open video generation models and the technical optimizations available in the Diffusers library to make these resource-intensive models accessible on consumer hardware. The primary takeaway is that while high-quality video generation remains computationally expensive, a combination of quantization, offloading, and chunked inference can drastically reduce VRAM requirements without severe compromises to speed.

The Landscape of Video Generation Models

Video generation is currently split between proprietary closed-source models and a growing ecosystem of open-source alternatives.

Closed-Source Models

Major industry players provide highly capable but proprietary models, including:

  • OpenAI: Sora
  • Google: Veo 2
  • Meta: MovieGen
  • RunwayML: Gen 3 Alpha
  • Pika Labs: Pika 2.0
  • KlingAI: Kling
  • Haliluo: MiniMax

Open-Source Models

Several open models have emerged to provide community access to video generation, including:

  • Tencent: Hunyuan Video
  • Genmo: Mochi-1
  • RhymesAI: Allegro
  • Lightricks: LTX Video
  • THUDM: CogVideoX

Technical Challenges and Limitations

Video generation is significantly more complex than image generation because it requires maintaining spatio-temporal consistency and coherence across frames, in addition to realism, aesthetics, and adherence to input conditions.

Key Limitations

  • High Resource Requirements: The cost of dataset collection, hardware, and training iterations makes open-source development expensive.
  • Generalization: Some open models struggle with out-of-distribution data or require highly specific, detailed prompting (e.g., LTX-Video) to achieve high-quality results.
  • Latency: High computational and memory demands lead to significant generation latency, which often acts as a barrier for local deployment.

Architecture of Open Video Models

Modern open video generation models typically follow a structure similar to text-to-image models but with 3D processing capabilities:

  • Text Encoders: Most models prefer T5, though HunYuan utilizes both CLIP-L and LLaMa 3.
  • Denoising Network: Based on the DiT (Diffusion Transformer) architecture with elements from PixArt, processing 3D video tokens that capture both spatial and temporal data.
  • Encoder-Decoder: Employs both spatial and temporal compression to convert between pixel and latent space. Frame-by-frame decoding is often used to manage memory.
  • Scheduler: A non-parametric scheduler manages timestep calculations and the denoising process.

Memory Optimization in Diffusers

Running state-of-the-art video models often requires staggering amounts of VRAM. For example, using standard torch.bfloat16 settings for a $121 \times 512 \times 768$ resolution video, the baseline memory requirements are:

  • HunyuanVideo: 60.09 GB
  • CogVideoX (1.5 5B): 36.51 GB
  • LTX-Video: 17.75 GB

Suite of Optimizations

To bring these models to consumer hardware, Diffusers provides several opt-in optimization categories:

  1. Quantization: Reducing weight precision via backends like bitsandbytes, torchao, and GGUF.
  2. Offloading: Using enable_model_cpu_offload() or enable_sequential_cpu_offload() to move layers to the CPU when not in active computation.
  3. Chunked Inference: Reducing activation state overhead through feed-forward chunking, decoder tiling/slicing, and split attention inference.
  4. Re-use of States: Skipping certain denoising steps by re-using past attention and MLP states.

Impact of Optimizations on HunyuanVideo

Hugging Face demonstrated that by chaining these techniques, the VRAM requirement for HunyuanVideo can be reduced from 60.10 GB (BF16 Base) to 6.56 GB using a combination of FP8 Upcasting, Group offload (leaf), and VAE tiling. Further reductions to ~5 GB are possible using Flash Attention and Optimized Feed-Forward.

Training and Fine-Tuning

Distillation Techniques

To improve inference speed, two primary distillation methods are used:

  • Timestep Distillation: Teaching the model to predict final outputs in fewer steps (e.g., Flux.1-Schnell, FastHunyuan).
  • Guidance Distillation: Reducing the two forward passes required by Classifier-Free Guidance into a single pass to halve generation time (e.g., HunyuanVideo, Flux.1-Dev).

Fine-Tuning with finetrainers

Hugging Face introduced the finetrainers repository to simplify the fine-tuning of open video models. This allows users to apply techniques like LoRA to specific models (e.g., CogVideoX) to emulate specific visual effects, such as a "dissolve" effect.

Sources