Simplifying, stabilizing, and scaling continuous-time consistency models

OpenAI has introduced sCM, a new approach to continuous-time consistency models that enables high-quality sample generation in only two sampling steps. This method provides a ~50x wall-clock speedup compared to traditional diffusion models while maintaining comparable sample quality.

Accelerated Sampling via sCM

sCM significantly reduces the computational overhead of generative AI by converting noise directly into noise-free samples. While traditional diffusion models require dozens or hundreds of sequential denoising steps to generate a single image, sCMs are designed to bypass this gradual process.

Key performance metrics for the largest sCM model (1.5 billion parameters) include:

  • Sampling Speed: Generates a single sample in 0.11 seconds on a single A100 GPU without inference optimization.
  • Efficiency: Uses less than 10% of the effective sampling compute required by other state-of-the-art generative models to achieve similar Fréchet Inception Distance (FID) scores.
  • Step Reduction: Achieves quality comparable to leading diffusion models using only two sampling steps.

Technical Implementation and Scaling

sCM simplifies the theoretical formulation of continuous-time consistency models, which stabilizes the training process and allows the models to scale to larger datasets and parameter counts. OpenAI scaled sCM training to 1.5 billion parameters on ImageNet at a 512×512 resolution.

Knowledge Distillation

The sCM approach distills knowledge from a pre-trained teacher diffusion model. OpenAI's research indicates that sCMs improve proportionally as the teacher diffusion model scales up. The relative difference in sample quality (measured by the ratio of FID scores) remains consistent across several orders of magnitude in model sizes, meaning the absolute quality gap diminishes as the models scale.

Quality vs. Steps

Increasing the number of sampling steps beyond two further reduces the quality gap between the sCM and the teacher diffusion model. Even at two steps, the relative difference in FID scores is less than 10% compared to the teacher model, which typically requires hundreds of steps.

Limitations and Constraints

Despite the speed gains, sCMs have specific technical dependencies and evaluation challenges:

  • Dependency on Teacher Models: The most effective sCMs currently rely on pre-trained diffusion models for both initialization and distillation.
  • Quality Gap: A small but consistent gap in sample quality remains between the sCM and its teacher diffusion model.
  • Metric Limitations: OpenAI notes that FID scores do not always perfectly reflect actual perceived sample quality, suggesting that sCM quality may need to be assessed differently based on specific application requirements.

Sources