Tencent-Hunyuan/HY-SOAR

What it solves

HY-SOAR addresses "exposure bias" in diffusion models. In standard training (SFT), models learn from ideal data paths, but during actual image generation (inference), they rely on their own previous predictions. If a model makes a small mistake early on, those errors compound, leading to poor final image quality.

How it works

Instead of using external reward models or human preference labels, SOAR teaches the model to correct its own trajectory errors in real-time during the denoising process. It does this by:

1. Sampling on-trajectory states: It performs a rollout step using the current model to see where it drifts.
2. Re-noising: It pushes the drifted state back toward the original noise endpoint to create auxiliary training states.
3. Analytical Correction: It supervises the denoiser using a specific geometric target (correction velocity) that points back toward the clean image.

This creates a dense, reward-free training signal that allows the model to co-evolve its training distribution with its actual inference behavior.

Who it’s for

Researchers and developers working with rectified-flow diffusion models (such as Stable Diffusion 3.5) who want to improve image alignment, text rendering, and aesthetic quality without the need for expensive reward models or preference datasets.

Highlights

• Reward-Free: No reward models, preference labels, or negative samples are required for training.
• Exposure-Bias Correction: Directly fixes the mismatch between how models are trained and how they actually generate images.
• On-Policy Supervision: The training distribution updates as the model improves, rather than staying fixed to static data.
• Compatible: Can replace standard SFT as a first post-training stage and remains compatible with subsequent RL alignment.