SalesforceAIResearch/DiffusionDPO

Code for "Diffusion Model Alignment Using Direct Preference Optimization"

What it solves

This project provides the training code for Diffusion-DPO, a method to align diffusion models (like Stable Diffusion) with human preferences. It allows developers to fine-tune image generation models so they produce images that better match what users actually prefer, rather than just relying on standard training objectives.

How it works

The system implements Direct Preference Optimization (DPO) for diffusion models. It takes a pretrained model (such as Stable Diffusion 1.5 or SDXL) and fine-tunes it using a dataset of preferred vs. non-preferred images. It includes tools for AI feedback via scoring models like PickScore, HPS, Aesthetics, and CLIP to evaluate the quality and alignment of the generated images.

Who it’s for

Researchers and machine learning engineers who want to align image generation models with human preferences using DPO, or those looking to replicate the results of the academic paper "Diffusion Model Alignment Using Direct Preference Optimization."

Highlights

  • Support for Multiple Models: Compatible with Stable Diffusion 1.5 and Stable Diffusion XL (SDXL).
  • AI Feedback Integration: Built-in support for scoring models including PickScore, HPS, Aesthetics, and CLIP.
  • Flexible Training: Includes scripts for both DPO and standard Supervised Fine-Tuning (SFT).
  • Pretrained Checkpoints: Provides access to DPO-aligned versions of SD1.5 and SDXL.