Diffusers 0.3 release adds image‑to‑image, textual inversion, inpainting, GPU optimizations, Mac MPS, ONNX support and new docs

TL;DR

Diffusers 0.3 brings image‑to‑image generation, textual inversion, experimental inpainting, memory‑efficient inference for modest GPUs, native Mac M1/M2 support, an ONNX export pipeline, and a comprehensive docs overhaul, dramatically lowering the barrier to use Stable Diffusion and inspiring a burst of community tools.

Image‑to‑Image Pipeline

The new StableDiffusionImg2ImgPipeline lets users supply an initial image and a text prompt to generate a transformed image. The API mirrors the text‑to‑image pipeline, adding init_image and a strength parameter that controls how much of the original image is preserved. Example usage:

from diffusers import StableDiffusionImg2ImgPipeline
import torch

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16",
    torch_dtype=torch.float16,
    use_auth_token=True,
)

init_image = preprocess(your_image)
prompt = "A fantasy landscape, trending on artstation"
images = pipe(
    prompt=prompt,
    init_image=init_image,
    strength=0.75,
    guidance_scale=7.5,
    generator=torch.Generator().manual_seed(42),
)["sample"]

A ready‑to‑run Space demo is also provided for quick experimentation.

Textual Inversion

Textual Inversion enables the creation of new concepts from as few as 3‑5 personal images. Users train a token embedding that can later be invoked in prompts, and the resulting concepts are shareable via the sd-concepts-library hub. The workflow includes:

  • Navigator Colab – browse over 150 community‑generated concepts.
  • Training Colab – fine‑tune a new token on a custom image set.
  • Inference Colab – generate images using the learned token.

Within days of release, the community contributed more than 200 concepts.

Experimental Inpainting Pipeline

The StableDiffusionInpaintPipeline accepts an image, a binary mask, and a prompt to replace masked regions while preserving the surrounding context. Example code:

from diffusers import StableDiffusionInpaintPipeline

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16",
    torch_dtype=torch.float16,
    use_auth_token=True,
).to("cuda")

images = pipe(
    prompt=["a cat sitting on a bench"] * 3,
    init_image=init_image,
    mask_image=mask_image,
    strength=0.75,
    guidance_scale=7.5,
).images

The feature is marked experimental, indicating ongoing improvements.

Optimizations for Smaller GPUs

Version 0.3 reduces VRAM consumption dramatically; Stable Diffusion now runs in ~3.2 GB of memory with only a ~10 % speed penalty. The key knob is attention slicing, enabled with:

from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16",
    torch_dtype=torch.float16,
    use_auth_token=True,
)
pipe = pipe.to("cuda")
pipe.enable_attention_slicing()

This optimization widens access to diffusion models on consumer‑grade GPUs.

Diffusers on macOS (M1/M2) via PyTorch MPS

Native support for Apple Silicon is added through the PyTorch mps device. Users can run Stable Diffusion on M1/M2 Macs with minimal code changes:

from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    use_auth_token=True,
)
pipe = pipe.to("mps")

prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt).images[0]

The official docs provide performance benchmarks and setup instructions.

Experimental ONNX Exporter and Pipeline

An ONNX‑based pipeline (StableDiffusionOnnxPipeline) enables inference on any ONNX‑compatible hardware, including CPUs. Usage example:

from diffusers import StableDiffusionOnnxPipeline

pipe = StableDiffusionOnnxPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="onnx",
    provider="CPUExecutionProvider",
    use_auth_token=True,
)

prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt).images[0]

A conversion script (convert_stable_diffusion_checkpoint_to_onnx.py) is also supplied for custom checkpoint export.

New Documentation Release (v0.3.0)

A dedicated docs sprint produced the first full‑version documentation site, covering:

  • Optimization techniques – FP16, attention slicing, and MPS guidance.
  • Training overview – high‑level steps for fine‑tuning diffusion models.
  • Contributing guide – how to submit patches and extensions.
  • API reference – detailed pages for schedulers and pipelines.

The docs are hosted at https://huggingface.co/docs/diffusers/v0.3.0/en/ and welcome community contributions.

Community Highlights

The release sparked a wave of community projects built on top of Diffusers:

  • Stable Diffusion Videos – tools for latent‑space interpolation and prompt morphing, available as a pip package and Colab notebooks.
  • Diffusers Interpret – an explainability suite that visualizes diffusion steps and token‑level attribution.
  • Japanese Stable Diffusion – a model trained on 100 M Japanese‑captioned images to capture cultural nuances.
  • Waifu Diffusion – a fine‑tuned checkpoint for high‑quality anime‑style generation.
  • Cross‑Attention Control – a repository allowing users to edit attention maps to modify prompt influence, replace objects, or inject styles.
  • Reusable Seeds – notebooks demonstrating how to reuse a seed from one generation to guide another, enabling controlled variations.

Getting Involved

The Diffusers repository remains open‑source; users are encouraged to star the GitHub project, join the Hugging Face Discord, and submit issues or pull requests. Continuous community contributions are essential to the library’s rapid evolution.

Sources