Stanford CS229 Lecture 11 (Spring 2026): Diffusion Models – Core Concepts and Training

TL;DR

Diffusion models generate high‑quality images by learning to reverse a fixed Gaussian noising process; they outperform GANs and VAEs and are trained with an ELBO‑style loss that treats intermediate noisy images as latent variables.

1. What a Diffusion Model Is

  • Goal: Given a dataset of natural images (distribution (p_{data})), learn a generative model (p_\theta) that can sample new images indistinguishable from the training set.
  • Key Idea: Start from pure Gaussian noise (x_T) and iteratively denoise it using a learned reverse Markov chain (p_\theta(x_{t-1}\mid x_t)) until a clean image (x_0) is produced.
  • Why It Matters: The approach replaces GANs and variational auto‑encoders (VAEs) as the dominant image‑generation technique because it yields more stable training and higher sample fidelity.

2. Forward (Noising) Process

  • The forward process (q) is fixed and adds Gaussian noise in small steps: [ x_t = \sqrt{1-\beta_t},x_{t-1} + \sqrt{\beta_t},\epsilon_t,\quad \epsilon_t \sim \mathcal N(0, I) ]
  • (\beta_t) is a small scalar (e.g., (10^{-2}) to (10^{-4})). Repeating this for (t=1\dots T) yields a distribution that converges to a standard normal as (T) grows.
  • The cumulative product (\bar\alpha_t = \prod_{i=1}^t (1-\beta_i)) shrinks the original image’s signal; as (t\to\infty), (\bar\alpha_t\to0) and (x_T) becomes pure Gaussian noise.

3. Reverse (Denoising) Process

  • The reverse dynamics are learned: a neural network (\mu_\theta(x_t, t)) predicts the mean of a Gaussian distribution (p_\theta(x_{t-1}\mid x_t) = \mathcal N(\mu_\theta(x_t, t), \sigma_t^2 I)).
  • The variance (\sigma_t^2) is usually fixed (derived from the forward schedule) rather than learned.
  • Although the forward process is deterministic given (x_{t-1}), the reverse must remain stochastic because multiple forward trajectories can lead to the same noisy state (x_t). Hence a Gaussian is a natural choice, and in the continuous‑time limit the true reverse process is provably Gaussian (classical results on stochastic differential equations).

4. Training Objective – ELBO / Variational Lower Bound

  • Latent Variable View: Treat the whole noisy trajectory (x_{1:T}) as latent variables (z) for the observed clean image (x_0).
  • ELBO Derivation: [ \log p_\theta(x_0) \ge \mathbb E_{q(x_{1:T}\mid x_0)}\big[\log p_\theta(x_{0:T}) - \log q(x_{1:T}\mid x_0)\big] ] This expands to a sum of KL divergences between the true forward conditionals (q(x_{t-1}\mid x_t, x_0)) and the learned reverse conditionals (p_\theta(x_{t-1}\mid x_t)), plus a reconstruction term for (x_0).
  • Chain‑Rule for KL: Using the Markov property, the KL between joint trajectories decomposes into a sum of per‑step KLs, each comparing two Gaussians. The per‑step loss simplifies to a weighted squared error between the predicted mean (\mu_\theta) and the analytically known posterior mean (\tilde\mu_t(x_t, x_0)).
  • Practical Loss: Most implementations use a simplified noise‑prediction loss: [ L_t = |\epsilon_t - \epsilon_\theta(x_t, t)|^2, ] where (\epsilon_\theta) predicts the injected noise. This is mathematically equivalent to the KL‑based ELBO up to a constant.

5. Why Not One‑Shot Denoising?

  • Directly mapping (x_T) to (x_0) would require learning a highly non‑linear function that collapses all information in a single step, making optimization difficult.
  • The gradual, multi‑step reverse chain provides smoother gradients and a well‑behaved objective, analogous to the benefits of curriculum learning.

6. Connections to Other Generative Models

  • GANs/VAEs: Both require adversarial or encoder‑decoder training. Diffusion models avoid adversarial instability and do not need a learned encoder; the forward process is analytically defined.
  • Autoregressive Models: Diffusion inference can be parallel across all pixels (or patches) for a given timestep, offering potential speed advantages over strictly sequential autoregressive sampling.
  • Language & Robotics: Recent work extends diffusion to text generation and robot action planning by treating token sequences or action trajectories as continuous data and applying the same forward‑reverse framework.

7. Continuous‑Time Perspective (Optional)

  • In the limit (\Delta t \to 0), the forward process becomes a stochastic differential equation (SDE): [ d x_t = -\frac{1}{2}\beta(t) x_t,dt + \sqrt{\beta(t)},dW_t, ] where (W_t) is Brownian motion.
  • The reverse SDE has the same diffusion term but a drift that depends on the score function (\nabla_{x_t}\log p_t(x_t)). Parameterizing the drift with a neural network recovers the discrete‑time reverse chain.

8. Practical Takeaways

  • Model Architecture: Typically a U‑Net or transformer‑based backbone predicts (\epsilon_\theta) for each timestep.
  • Training Schedule: Choose a noise schedule (\beta_1\dots\beta_T) (linear or cosine) that balances signal preservation early on and rapid diffusion later.
  • Sampling Speed: Recent research reduces the number of reverse steps from thousands to as few as 4–10 by learning better schedules or using distillation, but the original formulation uses thousands of steps.
  • Implementation Resources: Stanford’s lecture notes (linked in the video description) contain the full derivations and pseudocode.

This post follows the transcript of the Stanford CS229 Spring 2026 lecture on diffusion models. All equations and claims are taken directly from the instructor’s exposition; no external facts were added.

Sources