OpenAI Announces Improved Techniques for Training Consistency Models

TL;DR

OpenAI released a set of training improvements that remove the need for diffusion‑model distillation in consistency models, yielding single‑step FID scores of 2.51 on CIFAR‑10 and 3.25 on ImageNet 64×64—approximately 3.5×–4× better than previous consistency‑training results.

Background: Consistency Models and Their Limitations

Consistency models generate data in a single inference step, avoiding the iterative sampling of diffusion models. Historically, they have relied on distillation from a pre‑trained diffusion model and on learned perceptual metrics such as LPIPS to guide training. These dependencies constrain the final sample quality to the teacher diffusion model’s performance and introduce evaluation bias from LPIPS.

Direct‑From‑Data Training Eliminates Distillation

OpenAI’s new approach trains consistency models directly on raw data rather than distilling from a diffusion teacher. By removing the teacher‑student pipeline, the model is no longer capped by the diffusion model’s quality, allowing it to surpass previous consistency‑model limits.

Fixing a Theoretical Flaw: Removing EMA from the Teacher

During analysis, the authors identified a previously overlooked flaw: the Exponential Moving Average (EMA) applied to the teacher consistency model interferes with the consistency objective. The solution is to eliminate EMA from the teacher, which stabilizes training and improves convergence.

Replacing LPIPS with Pseudo‑Huber Loss

LPIPS, while useful for perceptual similarity, biases the training objective. OpenAI substitutes it with the Pseudo‑Huber loss, a robust‑statistics loss that combines the smoothness of L2 near zero error with the outlier resistance of L1. This change yields more reliable gradients and better visual fidelity.

Lognormal Noise Schedule and Step Doubling

Two additional training tricks are introduced:

  1. Lognormal noise schedule – the variance of the injected noise follows a lognormal distribution, better matching the data manifold’s scale across timesteps.
  2. Dynamic discretization steps – the total number of discretization steps is doubled after a fixed number of training iterations, effectively increasing the resolution of the consistency objective as training progresses.

Both adjustments help the model learn finer-grained consistency relationships.

Hyperparameter Tuning and Overall Gains

Comprehensive hyperparameter searches complement the architectural changes. The combined effect produces FID scores of 2.51 (CIFAR‑10) and 3.25 (ImageNet 64×64) in a single sampling step, representing 3.5× and 4× improvements over prior consistency‑training baselines.

Two‑Step Sampling Further Improves Quality

When the model is allowed a second sampling step, FID scores improve to 2.24 (CIFAR‑10) and 2.77 (ImageNet 64×64). These results surpass those obtained via distillation in both one‑step and two‑step regimes, narrowing the performance gap between consistency models and the broader class of state‑of‑the‑art generative models.

Implications for Generative Modeling

  • Efficiency – One‑step sampling retains the speed advantage of consistency models while achieving quality comparable to multi‑step diffusion.
  • Simplicity – Removing distillation simplifies the training pipeline, reducing computational overhead and eliminating the need for a large diffusion teacher.
  • Benchmark Advancement – The reported FID scores set a new state‑of‑the‑art for consistency models on standard image synthesis benchmarks.

Future Directions

OpenAI’s findings suggest further research avenues, such as extending the lognormal schedule to higher‑resolution datasets, exploring alternative robust losses, and integrating consistency training with conditional generation tasks.

All figures and claims are taken directly from OpenAI’s announcement dated 2024‑06‑20.

Sources