Stable Diffusion 3 Medium Integration with Diffusers
Hugging Face has integrated Stable Diffusion 3 Medium, a 2B parameter model from Stability AI, into the diffusers library. This release enables users to run high-quality text-to-image and image-to-image synthesis using a new architectural approach called Multimodal Diffusion Transformer (MMDiT) and a rectified flow-matching training objective.
Architectural Innovations in Stable Diffusion 3
Stable Diffusion 3 (SD3) departs from previous text-to-image architectures by utilizing a two-way flow of information between text and image data.
Multimodal Diffusion Transformer (MMDiT)
SD3 employs a novel MMDiT model alongside three text encoders: CLIP L/14, OpenCLIP bigG/14, and T5-v1.1-XXL. It also uses a 16-channel AutoEncoder similar to the one found in Stable Diffusion XL.
Unlike previous models that used cross-attention with fixed text representations, the MMDiT blocks process text inputs and pixel latents as sequences of embeddings. These sequences are embedded to a common dimensionality using separate sets of weights, concatenated, and passed through modulated attentions and MLPs. This allows both modalities to influence each other during the attention operation.
Rectified Flow Matching
SD3 is trained using a conditional flow-matching objective. This approach defines the forward noising process as a "rectified flow" that connects data and noise distributions via a straight line.
To support this, Hugging Face introduced the FlowMatchEulerDiscreteScheduler. This scheduler implements Euler method steps and includes a shift parameter for resolution-dependent timestep schedule shifting. For the 2B model, a shift=3.0 is recommended to better handle noise scaling at higher resolutions.
Memory Optimization Techniques
Because SD3 utilizes the large T5-XXL text encoder (4.7B parameters), running the model on GPUs with less than 24GB of VRAM is challenging. Hugging Face provides several optimization strategies to reduce the memory footprint:
- Model Offloading: Using
pipe.enable_model_cpu_offload()moves model components to the CPU when not in use, reducing VRAM usage at the cost of increased latency. - Dropping the T5 Encoder: Removing the T5-XXL encoder during inference (
text_encoder_3=None) significantly reduces memory requirements with a slight loss in performance. - 8-bit Quantization: Loading the T5-XXL model in 8-bit precision via the
bitsandbyteslibrary further lowers memory consumption.
Memory Benchmark Comparison
Benchmarks conducted on an A100 GPU (80GB VRAM) using fp16 precision and PyTorch 2.3 show the following trade-offs:
| Technique | Inference Time (secs) | Memory (GB) |
|---|---|---|
| Default | 4.762 | 18.765 |
| Offloading | 32.765 | 12.0645 |
| Offloading + no T5 | 19.110 | 4.266 |
| 8bit T5 | 4.932 | 10.586 |
Performance and Fine-tuning
Inference Acceleration with torch.compile()
Users can achieve significant speedups by using torch.compile() to optimize the compute graphs of the VAE and transformer components. In benchmarks on a single 80GB A100, the average inference time was reduced to 0.585 seconds, representing a 4X speedup over eager execution.
DreamBooth and LoRA Training
Hugging Face has released a DreamBooth fine-tuning script for SD3 that leverages Low-Rank Adaptation (LoRA). This script allows for efficient model customization and serves as a reference for implementing training pipelines based on rectified flow.