Hierarchical Text-Conditional Image Generation with CLIP Latents
OpenAI has introduced a hierarchical approach to text-conditional image generation that leverages CLIP (Contrastive Language-Image Pre-training) latents. By decoupling the process into a prior and a decoder, the model improves image diversity and enables advanced capabilities like zero-shot image manipulation.
Two-Stage Generation Architecture
The system operates as a two-stage model designed to utilize the robust semantic and style representations learned by CLIP.
The Prior
The first stage is the prior, which generates a CLIP image embedding given a text caption. OpenAI experimented with both autoregressive and diffusion models for this component. The researchers found that diffusion models for the prior are computationally more efficient and produce higher-quality samples than autoregressive versions.
The Decoder
The second stage is the decoder, which takes the image embedding produced by the prior and generates a final image conditioned on that embedding. The researchers utilized diffusion models for the decoder stage.
Key Technical Improvements and Capabilities
Enhanced Image Diversity
Explicitly generating image representations (CLIP latents) first improves the diversity of the generated images with minimal loss in photorealism and similarity to the original text caption.
Semantic and Style Preservation
Because the decoder is conditioned on a CLIP image representation, it can produce variations of an image that preserve the core semantics and style of the original while varying non-essential details that are not captured in the image representation.
Zero-Shot Image Manipulation
The joint embedding space of CLIP allows for language-guided image manipulations in a zero-shot fashion. This means the model can alter images based on text prompts without requiring specific training for those particular edits.
Summary of Model Components
| Component | Model Type Used | Primary Function |
|---|---|---|
| Prior | Diffusion Model | Text Caption $\rightarrow$ CLIP Image Embedding |
| Decoder | Diffusion Model | CLIP Image Embedding $\rightarrow$ Final Image |