Glow: Better Reversible Generative Models
OpenAI has introduced Glow, a reversible generative model that utilizes invertible 1x1 convolutions to synthesize high-resolution images. This architecture simplifies previous flow-based models, enabling exact latent-variable inference, efficient parallel sampling, and the discovery of semantic features for data manipulation.
Advantages of Flow-Based Generative Models
Glow belongs to the class of flow-based generative models, which offer several technical advantages over Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs):
- Exact Inference and Likelihood: Unlike VAEs, which provide approximate latent variable inference, or GANs, which lack an encoder entirely, reversible models allow for exact latent-variable inference and the optimization of exact log-likelihood of the data.
- Efficient Parallelism: While autoregressive models (such as PixelCNN) are reversible, their synthesis is difficult to parallelize. Glow is efficient to parallelize for both inference and synthesis.
- Manipulatable Latent Space: Reversible models provide a latent space where datapoints can be directly represented, facilitating interpolations and meaningful modifications of existing datapoints. This is more robust than GANs, which may not have full support over the data distribution.
- Memory Efficiency: Based on the principles of RevNet, computing gradients in reversible neural networks requires a constant amount of memory relative to depth, rather than linear memory.
Technical Contributions and Architecture
Glow improves upon the RealNVP architecture by simplifying the model and introducing a new invertible operation.
Invertible 1x1 Convolutions
The primary contribution of Glow is the replacement of fixed permutations with learned 1x1 convolution operations. In previous models, channel-wise masking relied on rigid permutations to shuffle data. Because permuting channels is a special case of a linear transformation, Glow uses 1x1 convolutions with an equal number of input and output channels to learn the optimal permutation. These weights are initialized as random rotation matrices and optimized efficiently using LU decomposition.
Architectural Simplifications
- Removal of Checkerboard Masking: Glow removes the layers with checkerboard masking used in RealNVP to simplify the overall architecture.
- Activation Normalization: The model replaces batch normalization with an activation normalization layer. This layer shifts and scales activations using data-dependent initialization, which allows the model to scale up in size and function with minibatch sizes as small as one, which is necessary for processing large images.
Performance and Results
Glow demonstrates significant quantitative improvements over RealNVP across multiple benchmarks, measured in bits per dimension:
| Dataset | RealNVP | Glow |
|---|---|---|
| CIFAR-10 | 3.49 | 3.55 |
| Imagenet 32x32 | 4.28 | 4.09 |
| Imagenet 64x64 | 3.98 | 3.81 |
| LSUN (bedroom) | 2.72 | 2.38 |
| LSUN (tower) | 2.81 | 2.46 |
| LSUN (church outdoor) | 3.08 | 2.67 |
Image Synthesis and Latent Manipulation
Training on a dataset of 30,000 high-resolution faces, Glow can generate 256 x 256 samples in approximately 130ms on an NVIDIA 1080 Ti GPU. The researchers found that scaling the standard deviation of the latents by a temperature of 0.7 often improved sample quality.
Because Glow possesses a perfect encoder, it can be used for unsupervised attribute manipulation. By encoding images with and without a specific attribute (e.g., blonde hair) and calculating the difference between their average latent vectors, a "manipulation vector" is created. This vector can then be added to the latent representation of any input image to modify that specific attribute without requiring labels during the initial training phase.
Scale and Future Research
To train models with over 100 million parameters, OpenAI utilized Horovod across a cluster of five machines, each equipped with eight GPUs, and employed gradient checkpointing to manage memory.
Future research directions identified by the team include:
- Likelihood Competitiveness: Combining flow-based models with VAEs and autoregressive models to match their log-likelihood performance while retaining efficient sampling.
- Computational Efficiency: Exploring self-attention architectures or progressive training to reduce the need for the current depth (
600 convolution layers) and parameter count (200M) required for high-resolution images.