tdrussell/diffusion-pipe

A pipeline parallel training script for diffusion models.

What it solves

It enables the training of massive diffusion models that are too large to fit on a single GPU. By utilizing pipeline parallelism, it allows users to distribute the model's workload across multiple GPUs, making high-end model training accessible on more limited hardware.

How it works

The project uses a hybrid of data and pipeline parallelism powered by DeepSpeed. It splits the model into stages across GPUs and can run multiple instances of the model simultaneously. To optimize memory and speed, it pre-caches latents and text embeddings to disk using the Huggingface Datasets library, removing the need to keep VAEs and text encoders in VRAM during the actual training process.

Who it’s for

Developers and AI researchers who want to train or fine-tune large-scale image and video diffusion models (such as Flux, SDXL, and HunyuanVideo) using multi-GPU setups.

Highlights

  • Broad Model Support: Compatible with a vast array of models including SDXL, Flux, LTX-Video, HunyuanVideo, Cosmos, and many others.
  • Unified Framework: Supports both image and video models within a single pipeline.
  • Memory Efficiency: Includes VRAM-reducing features like AdamW8BitKahan optimizer, block swapping, and Unsloth activation checkpointing.
  • Flexible Training: Supports training LoRAs directly on quantized models and provides full state checkpointing for resuming interrupted sessions.

Related

  • Dispatch
    OpenAI Techniques for Training Large Neural Networks

    OpenAI outlines the primary parallelism strategies—data, pipeline, tensor, and expert parallelism—and memory-saving techniques used to orchestrate GPU clusters for training massive AI models.

  • Project
    hpcaitech/ColossalAI

    A distributed deep learning framework that makes training and inference for large AI models faster and cheaper through advanced parallelism and memory management.

  • Project
    NVIDIA-NeMo/Curator

    A GPU-accelerated data curation framework for building repeatable pipelines to load, filter, deduplicate, and transform massive text, image, video, and audio datasets for AI training.

  • Project
    modelscope/DiffSynth-Studio

    An open-source diffusion model engine that enables high-performance inference and training of image, video, and audio generative models on consumer-grade GPUs.

  • Project
    baidu-baige/LoongForge

    A high-performance training framework for LLMs, VLMs, diffusion, and embodied models that optimizes parallelism and kernel efficiency to reduce training costs.