NVlabs/FastGen
NVIDIA FastGen: Fast Generation from Diffusion Models
What it solves
FastGen is designed to speed up the generation process of diffusion models. It provides a unified framework to apply various acceleration and distillation techniques to make generative models produce high-quality outputs faster, supporting large-scale training for models with 10B+ parameters.
How it works
The framework is network- and data-agnostic, allowing users to integrate their own architectures and datasets. It implements a wide range of distillation and training methods, including Consistency Models (CM, sCM, TCM, MeanFlow), Distribution Matching (DMD2, f-Distill, LADD, CausVid, Self-Forcing), Fine-Tuning (SFT, CausalSFT), and Knowledge Distillation (KD, CausalKD).
It leverages PyTorch and supports distributed training via DDP and FSDP2 for model sharding to handle massive models.
Who it’s for
AI researchers and developers working on generative AI, specifically those looking to accelerate diffusion-based image and video generation (T2I, I2V, V2V).
Highlights
- Broad Modality Support: Works across text-to-image, image-to-video, and video-to-video tasks.
- Extensive Method Library: Includes a comprehensive set of distillation and acceleration algorithms.
- Large-Scale Capability: Supports training models with 10 billion or more parameters using FSDP2.
- Flexible Architecture: Agnostic to the specific neural network or dataset used, with built-in support for models like SDXL, Flux, and CogVideoX.