yuanchenyang/smalldiffusion

Simple and readable code for training and sampling from diffusion models

What it solves

smalldiffusion is a lightweight library designed to simplify the process of training and sampling from diffusion and flow models. It removes the complexity of implementing these models from scratch, providing a concise core (under 100 lines of code) that is easy to extend for researchers and developers experimenting with new models or samplers.

How it works

The library operates through the interaction of three primary components:

  • Data: Uses PyTorch Dataset and DataLoader for training. It includes built-in support for 2D toy datasets, pixel-space images, and latent-space image datasets.
  • Model: Supports various architectures like MLP, U-Net, and Diffusion Transformer (DiT). It allows for different parameterizations, including score-prediction, flow-prediction, and data-prediction (x0).
  • Schedule: Manages the noise levels (sigmas) used during training and sampling. It provides several built-in schedules, such as LogLinear, DDPM, LDM (Stable Diffusion style), and Flow matching schedules.

Sampling is handled by a unified samples() generator that supports DDPM, DDIM, and other accelerated sampling algorithms by adjusting hyperparameters like gam and mu.

Who it’s for

Researchers and developers who want a clean, minimal implementation of diffusion models for experimentation, training on small-to-medium datasets (like CIFAR-10 or FashionMNIST), or sampling from pretrained Hugging Face diffusers models.

Highlights

  • Minimalist Core: Training and sampling logic is contained in less than 100 lines of code.
  • Versatile Model Support: Includes implementations of U-Net and Diffusion Transformer (DiT).
  • Flexible Parameterizations: Supports score, flow, and data prediction objectives.
  • Pretrained Model Integration: Includes a wrapper to use the library's samplers with pretrained Stable Diffusion models.
  • High Performance: Example training on ImageNet 256x256 using flow matching achieves an unconditional FID of around 27.