tdrussell/diffusion-pipe
A pipeline parallel training script for diffusion models.
What it solves
It enables the training of massive diffusion models that are too large to fit on a single GPU by utilizing pipeline parallelism. It provides a unified framework for training both image and video generation models, reducing the complexity of managing different architectures.
How it works
The project uses a hybrid of data and pipeline parallelism via DeepSpeed to distribute the model across multiple GPUs. To optimize VRAM usage, it implements pre-caching of latents and text embeddings to disk, allowing the VAE and text encoders to be unloaded during the actual training process. It also supports memory-saving techniques like 8-bit optimizers, block swapping, and activation checkpointing.
Who it’s for
AI researchers and developers who need to fine-tune or train large-scale image and video diffusion models (such as Flux, SDXL, or HunyuanVideo) and lack the VRAM of a single massive GPU.
Highlights
- Broad Model Support: Compatible with a wide array of modern models including SDXL, Flux, LTX-Video, HunyuanVideo, Cosmos, and Wan2.1.
- Pipeline Parallelism: Specifically designed to handle models larger than a single GPU's memory.
- Unified Image/Video Workflow: Supports both modalities in a single training script.
- Efficient Caching: Multi-process, multi-GPU pre-caching of embeddings and latents to speed up training and save memory.
- Training Management: Includes Tensorboard logging, evaluation on held-out sets, and state checkpointing for resuming training.