diffusers: a modular toolbox for running and training state-of-the-art diffusion models across images, audio, and 3D structures

What it solves

Diffusers is a library designed to provide easy access to state-of-the-art pretrained diffusion models for generating images, audio, and 3D molecular structures. It simplifies the process of both running inference (generating content) and training custom diffusion models, providing a modular toolbox that prioritizes usability and customizability over strict abstractions.

How it works

The library is built around three core components:

• Diffusion Pipelines: High-level APIs that allow users to run complex inference tasks with just a few lines of code.
• Noise Schedulers: Interchangeable components that control the diffusion speed and the quality of the output.
• Pretrained Models: Modular building blocks that can be combined with schedulers to create custom end-to-end diffusion systems.

Who it’s for

It is intended for developers and researchers who want to use pretrained diffusion models for tasks like text-to-image, image-to-image, inpainting, and super-resolution, as well as those looking to train their own diffusion models from scratch or fine-tune them.

Highlights

• Broad Modality Support: Supports image, audio, and 3D molecular structure generation.
• Extensive Model Hub: Access to over 30,000 checkpoints via the Hugging Face Hub.
• Modular Architecture: Allows users to swap schedulers and models to tweak system behavior.
• Optimization: Includes guides and tools for reducing memory consumption and increasing inference speed.
• Wide Adoption: Used by over 14,000 other GitHub repositories.