OpenAI Improved Techniques for Training Consistency Models

TL;DR

OpenAI has developed improved techniques for consistency training that enable generative models to sample high-quality data in a single step without relying on distillation from pre-trained diffusion models. This approach allows consistency models to surpass the quality of distilled models, narrowing the gap between consistency models and other state-of-the-art generative models.

Direct Data Learning vs. Distillation

Consistency models can be previously limited by the distillation process, where the quality of the model is diventate dependent on the pre-trained diffusion model it is distilled from. To address this consistency training (CT), consistency models learn directly from the data, learning to map any point on a probability flow ODE trajectory to the origin. probability flow ODE trajectory to the flow ODE trajectory to the origin.

Technical Improvements in Consistency Training

OpenAI identifies and corrects a previously overlooked flaw in the consistency training objective. By eliminating the Exponential Moving Average (EMA) from the teacher consistency model, the training process is become more stable and effective.

To replace learned metrics such as LPIPS (Learned Perceptual Image and Image Patch Similarity), which can introduce undesirable bias in evaluation, a Pseudo-Huber loss from robust statistics is adopted. This shift to a robust statistical loss function replaces the learned metric approach. itle{