modelscope/DiffSynth-Studio

Enjoy the magic of Diffusion models!

What it solves

DiffSynth-Studio is an open-source diffusion model engine designed to simplify the deployment and training of generative models. It addresses the high hardware requirements typically associated with large diffusion models by providing advanced memory management and quantization techniques, making these models accessible on consumer-grade GPUs.

How it works

The framework integrates a wide variety of mainstream open-source diffusion models across different modalities. It employs a dynamic scheduling system to move model parameters between disk, memory, and VRAM to optimize resource usage. To further reduce memory footprints, it supports parameter quantization (such as NF4 and INT8) and CPU offload training, which allows model weights to be moved layer-by-layer during the training process.

Who it’s for

This tool is intended for developers and researchers who want to run inference or train (including LoRA and Adapter models) on generative models for images, video, and audio, particularly those with limited hardware resources.

Highlights

  • Multi-modal Support: Integrates models for image, video, and audio generation, as well as image quality metrics.
  • VRAM Management: Dynamic scheduling of parameters across disk, memory, and VRAM to support low-VRAM GPUs.
  • Advanced Quantization: Supports NF4 and INT8 quantization to reduce VRAM requirements for both inference and training.
  • Flexible Training: Supports base model training, LoRA, and Adapter models, including a specialized "split training" process using a computational graph inference engine.
  • CPU Offload Training: Enables training of large models on consumer GPUs by moving weights between CPU and GPU layer-by-layer.

Related

  • Project
    huggingface/finetrainers

    A library for the accessible training and fine-tuning of diffusion models, focusing on memory-efficient text-to-video and text-to-image generation.

  • Project
    huggingface/diffusers

    A modular library for using and training state-of-the-art diffusion models to generate images, audio, and 3D molecular structures.

  • Project
    vitoplantamura/OnnxStream

    A memory-optimized inference library that enables large AI models like Stable Diffusion and LLMs to run on ultra-low-RAM devices by streaming weights and using attention slicing.

  • Dispatch
    State of Open Video Generation Models in Diffusers

    Hugging Face provides a comprehensive overview of open video generation models and introduces a suite of Diffusers optimizations that can reduce VRAM requirements for models like HunyuanVideo from 60GB to approximately 6.5GB.

  • Project
    NVIDIA/TensorRT-LLM

    An open-source optimization library from NVIDIA that maximizes inference performance, throughput, and efficiency for LLMs and visual generation models on NVIDIA GPUs.