Implicit Generation and Generalization Methods for Energy-Based Models

OpenAI has developed methods for the stable and scalable training of energy-based models (EBMs), resulting in improved sample quality and generalization capabilities. EBMs offer a flexible alternative to traditional generative models by assigning an unnormalized probability scalar, or "energy," to each input data point, allowing for implicit generation through iterative refinement.

Implicit Generation via Iterative Refinement

EBMs generate samples implicitly using an iterative refinement process based on Langevin dynamics, which involves performing noisy gradient descent on the energy function to reach low-energy configurations. This approach differs from GANs, VAEs, and Flow-based models because it does not require an explicit neural network to generate samples.

This implicit generation method provides three primary technical advantages:

Adaptive Computation Time: The model can run sequential refinement for short periods to produce coarse samples or for longer periods to generate sharp, diverse samples. In the limit of infinite time, this procedure is known to generate true samples from the energy model.
Freedom from Generator Network Constraints: Unlike VAEs and Flow-based models, which must learn a map from a continuous space to a potentially disconnected space of data modes, EBMs can easily assign low energies to disjoint regions.
Built-in Compositionality: Because EBMs represent unnormalized probability distributions, they can be naturally combined using a product of experts or other hierarchical structures.

Generative Capabilities and Applications

EBMs can produce qualitatively and quantitatively high-quality images, with quality increasing as the refinement process runs longer at test time. OpenAI demonstrated these capabilities through several applications:

Image Manipulation: The models can perform image auto-completion (inpainting) and morph images from one class to another (e.g., morphing a truck into a frog).
Robot Dynamics: EBMs can generate stable robot dynamics trajectories over many timesteps. While feedforward models often collapse to a mean prediction, EBMs can generate a diverse set of feasible future trajectories from the same starting state.

Generalization and Robustness

Energy-based models demonstrate superior performance in classification tasks involving out-of-distribution datasets compared to likelihood models such as Flow-based and autoregressive models.

Additionally, conditional EBMs exhibit strong generalization to adversarial perturbations. OpenAI noted that their model performed classification better than models explicitly trained against adversarial perturbations, despite the EBM not being specifically trained for classification.

Training Observations and Lessons Learned

OpenAI identified several challenges and observations during the training of EBMs:

HMC Limitations: Vanilla Hamiltonian Monte Carlo (HMC) was difficult to apply to EBM training because optimal step sizes and leapfrog simulation numbers vary significantly during training.
Ensemble Training: Training ensembles of energy functions provided some benefit but did not justify the added complexity.
Gradient Penalties: Adding a gradient penalty term was found to be unsuccessful, as it appeared to hinder both sampling and model capacity.

Future Directions: Compositionality

Preliminary results indicate that multiple EBMs can be composed via a product of experts model. In a 2D experiment, OpenAI trained one model on shapes of different sizes at a fixed position and another on shapes of the same size at different positions. By combining these models, the system could generate shapes of different sizes at different locations, even though it had never seen examples where both variables changed simultaneously.