Stable Diffusion 3.5 Large Integration with Diffusers

Hugging Face has integrated Stable Diffusion 3.5 (SD3.5) into the Diffusers library, providing access to an 8B parameter model designed to improve upon Stable Diffusion 3. This release includes two primary checkpoints: a standard 8B large model and a timestep-distilled 8B large model that enables high-quality image generation in significantly fewer steps.

Architectural Enhancements in SD3.5 Large

Stable Diffusion 3.5 Large maintains a transformer architecture similar to SD3 Medium but introduces two key technical changes to improve scaling and performance:

  • QK Normalization: SD3.5 Large implements QK normalization, a standard practice for training large transformer models to ensure stability.
  • Dual Attention Layers: The model replaces the single attention layers used for each modality stream in the MMDiT blocks with double attention layers.

Other components, including the VAE, text encoders, and noise scheduler, remain identical to those used in SD3 Medium.

Inference Implementation with Diffusers

SD3.5 is a gated model requiring users to accept the terms on the Hugging Face Hub and authenticate via huggingface-cli login before use. The recommended precision for inference is torch.bfloat16.

Standard and Turbo Inference

Users can implement the standard 8B model using the StableDiffusion3Pipeline. The timestep-distilled version, referred to as the "turbo" model, eliminates the need for classifier-free guidance and typically generates images in only 4 to 8 steps, significantly reducing latency.

Memory Optimization and Quantization

Due to the 8B parameter size, SD3.5 Large requires substantial memory. Diffusers supports bitsandbytes quantization to enable inference on consumer-grade hardware. By loading the transformer in "NF4" precision using BitsAndBytesConfig, users can significantly reduce the VRAM footprint. Additional memory savings can be achieved by calling pipeline.enable_model_cpu_offload().

Fine-Tuning and LoRA Training

It is possible to fine-tune SD3.5 Large on consumer GPUs with 24GB of VRAM by combining bitsandbytes and peft. While the existing SD3 training scripts are compatible, quantization requires specific modifications:

  1. Initialize the transformer using a quantization configuration or a pre-quantized checkpoint.
  2. Apply prepare_model_for_kbit_training() from the peft library to prepare the model for quantized training.

Flexible Model Loading

Diffusers provides the from_single_file method for the SD3Transformer2DModel, allowing users to load the transformer directly from the original .safetensors checkpoint files published by Stability AI, which can then be integrated into a StableDiffusion3Pipeline.

Sources