Introducing Würstchen: Fast Diffusion for Image Generation

TL;DR

Hugging Face has introduced Würstchen, a text-to-image diffusion model designed for extreme efficiency. By utilizing a novel two-stage compression architecture, Würstchen achieves 42x spatial compression, allowing it to generate high-resolution images faster and with significantly lower memory and training compute requirements than previous models like Stable Diffusion.

High-Compression Architecture

Würstchen reduces computational costs for training and inference by operating in a highly compressed latent space. While typical diffusion models use spatial compression in the 4x to 8x range, Würstchen employs a 42x spatial compression to enable training on smaller latent representations without sacrificing image detail.

This architecture consists of three primary stages:

  • The Decoder (Stages A and B): This component decodes compressed images back into pixel space. It consists of Stage A (a VQGAN) and Stage B (a Diffusion Autoencoder).
  • The Prior (Stage C): This model is learned within the highly compressed latent space. It handles the text-conditional component of the image generation process.

Performance and Efficiency Gains

Würstchen provides substantial improvements in inference speed and memory usage compared to models such as Stable Diffusion XL (SDXL). It is specifically designed to be accessible to users without high-end hardware like A100 GPUs.

Training Compute Reduction

The model demonstrates a drastic reduction in the GPU hours required for training:

  • Würstchen v1 (512x512): Required 9,000 GPU hours, representing a 16x reduction in cost compared to the 150,000 GPU hours spent on Stable Diffusion 1.4.
  • Würstchen v2 (up to 1536 resolution): Required 24,602 GPU hours, which is approximately 6x cheaper than SD1.4 despite training at higher resolutions.

Technical Implementation and Usage

Würstchen is fully integrated into the diffusers library, enabling it to be used via the AutoPipelineForText2Image interface. The model was trained on resolutions between 1024x1024 and 1536x1536, though it can produce quality outputs at 1024x2048 and can be cheaply fine-tuned for 2048x2048 resolutions.

Integrated Optimizations

Through its diffusers integration, Würstchen supports several out-of-the-box optimizations:

  • Memory Management: Model offloading and sequential CPU offload minimize VRAM usage.
  • Hardware Support: Support for the mps device on Apple Silicon Macs.
  • Prompting: Prompt weighting via the Compel library.
  • Reproducibility: Use of generators for consistent results.

Advanced Performance Tuning

Users can further accelerate Würstchen using two primary techniques:

  1. Flash Attention: The model automatically utilizes PyTorch 2.0's scaled_dot_product_attention (SDPA) for memory-efficient attention. Users on PyTorch 1.x can use the xFormers library via pipeline.enable_xformers_memory_efficient_attention().
  2. Torch Compile: Applying torch.compile to the prior and decoder models (using mode="reduce-overhead" and fullgraph=True) provides an additional performance boost after an initial compilation period of up to two minutes.

Sources