worldbench/DiffusionOPSD

What it solves

Diffusion reward optimization typically only provides a score after a full multi-step denoising process, creating a "supervision gap" where the model doesn't know how to improve intermediate predictions. DiffusionOPSD closes this gap by turning image-level rewards into explicit, intermediate targets for the model to follow during training.

How it works

The project implements an on-policy self-distillation loop:

1. Query Collection: A frozen "behavior policy" generates trajectories and selects low-noise query states.
2. Target Construction: The system calculates a clean-output anchor and uses differentiable reward gradients to create bounded positive (reward-ascent) and negative (reward-descent) targets around that anchor.
3. Finite Fitting: The trainable policy fits these detached targets, separating the quality of the target from the model's ability to realize it.
4. Refresh: An Exponential Moving Average (EMA) updates the behavior policy, and the process repeats with new trajectories.

Who it’s for

Researchers and developers working on reward-guided post-training for diffusion models who want to improve image quality and training efficiency.

Highlights

• On-policy supervision: Uses states actually visited by the current model rather than offline substitutes.
• Explicit targets: Converts scalar rewards into concrete positive and negative clean-output targets.
• High efficiency: Reduces training GPU-hours by 40% to 63% compared to DiffusionNFT.
• Broad compatibility: Supports SD3.5-M and Z-Image-Turbo backbones across seven different reward evaluators (e.g., PickScore, HPSv3, DeQA).
• Mixed-reward training: Ability to train using weighted sums of multiple reward models simultaneously.