hao-ai-lab/FastVideo

What it solves

FastVideo addresses the high computational cost and slow generation speeds associated with state-of-the-art video Diffusion Transformers (DiTs). It provides a unified framework to accelerate both the post-training (fine-tuning and distillation) and the real-time inference of video generation models.

How it works

FastVideo employs several optimization techniques to reduce latency and increase throughput:

• Post-Training Optimizations: It supports full and LoRA fine-tuning, Distribution Matching Distillation (DMD2) for stepwise reduction, and sparse distillation to achieve significant denoising speedups (over 50x).
• Attention Mechanisms: It implements Video Sparse Attention (VSA) and Sliding Tile Attention to reduce the complexity of the attention mechanism in video models.
• Inference Acceleration: The framework utilizes sequence parallelism for distributed inference across multiple GPUs and integrates multiple high-performance attention backends.
• Causal Distillation: It uses "Self-Forcing" to enable causal distillation for autoregressive models.

Who it’s for

This framework is designed for AI researchers and developers building high-performance video generation models who need to reduce inference latency or optimize training and distillation pipelines.

Highlights

• Real-time Generation: Includes "Dreamverse," a platform for real-time video streaming and "vibe directing" (interactive editing).
• High Efficiency: Capable of generating 5s 1080p video in 4.5s on a single GPU.
• Broad Hardware Support: Compatible with NVIDIA GPUs (H100, A100, 4090), Apple Silicon (MPS), and Linux, Windows, and MacOS.
• Scalable Training: Supports FSDP2, sequence parallelism, and selective activation checkpointing for large-scale model training.