Stable Diffusion with Diffusers

TL;DR

Hugging Face released a guide for running Stable Diffusion v1‑4 (and later versions) with the 🤗 Diffusers library, detailing installation, licensing, inference code, and the underlying latent diffusion architecture.

License requirements

The Stable Diffusion model is distributed under a license that (1) prohibits illegal or harmful generation, (2) grants users full rights to generated outputs while holding them accountable, and (3) permits commercial use and weight redistribution provided the same restrictions are passed to downstream users.

Quick start with Diffusers

  • Install the required packages:
pip install diffusers==0.10.2 transformers scipy ftfy accelerate
  • Load the pipeline (v1‑4 shown; other versions such as 1.5, 2, 2.1 work similarly):
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
pipe.to("cuda")
  • For GPUs with <10 GB memory, load the fp16 checkpoint:
pipe = StableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4", revision="fp16", torch_dtype=torch.float16
)
  • Generate an image:
prompt = "a photograph of an astronaut riding a horse"
image = pipe(prompt).images[0]
  • Deterministic output is achieved by seeding a torch.Generator and passing it to the pipeline.
  • The guidance_scale parameter (default 7.5) controls classifier‑free guidance; values between 7 and 8.5 are recommended.
  • The number of denoising steps (num_inference_steps) defaults to 50; fewer steps speed up generation at the cost of quality.
  • Non‑square images can be created by specifying height and width (both must be multiples of 8). Using 512 in one dimension and a larger multiple of 8 in the other yields the best results.

Understanding Stable Diffusion’s architecture

Stable Diffusion is a latent diffusion model that reduces memory and compute by operating in a compressed latent space (8×8 spatial reduction, i.e., a 512×512 image becomes a 64×64 latent tensor). It consists of three core components:

  1. Variational Auto‑Encoder (VAE) – encodes images to latents and decodes latents back to pixel space; only the decoder is needed at inference time.
  2. U‑Net – predicts the noise residual for each diffusion step; includes cross‑attention layers that condition on text embeddings.
  3. Text encoder – a pre‑trained CLIP text model (CLIPTextModel) that converts prompts into 77×768 embeddings; the encoder is frozen during training.

During inference, a random latent seed and the text embeddings are fed to the U‑Net, which iteratively denoises the latents over ~50 steps using a scheduler (default PNDM, with alternatives DDIM and K‑LMS). The final latent is then decoded by the VAE to produce a 512×512 image.

Customizing the pipeline

Advanced users can replace individual components (e.g., VAE, scheduler, UNet) by loading them from the model repository with the subfolder argument:

from diffusers import AutoencoderKL, UNet2DConditionModel, PNDMScheduler
vae = AutoencoderKL.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="vae")
unet = UNet2DConditionModel.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="unet")
scheduler = LMSDiscreteScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000)

The example in the blog shows a full inference loop using the K‑LMS scheduler, classifier‑free guidance, and manual latent handling.

Practical tips

  • Use torch.float16 on GPUs with limited memory.
  • Set height/width to multiples of 8 to avoid shape mismatches.
  • Increase guidance_scale for stronger prompt adherence, but expect reduced diversity.
  • Reduce num_inference_steps for faster results; increase for higher fidelity.
  • The pipeline returns a dictionary with images and an nsfw_content_detected flag.

Resources

Citation

@article{patil2022stable,
  author = {Patil, Suraj and Cuenca, Pedro and Lambert, Nathan and von Platen, Patrick},
  title = {Stable Diffusion with 🧨 Diffusers},
  journal = {Hugging Face Blog},
  year = {2022},
  note = {https://huggingface.co/blog/stable_diffusion}
}

Sources