DALL·E: Creating images from text

OpenAI has announced DALL·E, a neural network capable of generating original images from natural language descriptions. By leveraging a 12-billion parameter transformer architecture, DALL·E can synthesize complex scenes, combine unrelated concepts, and perform zero-shot visual reasoning.

Architecture and Technical Approach

DALL·E is a decoder-only transformer language model that processes text and images as a single stream of data. The model is trained using maximum likelihood to generate tokens autoregressively.

Tokenization and Data Stream

  • Input/Output Stream: The model handles a total of 1,280 tokens per sequence, consisting of 256 tokens for the text prompt and 1,024 tokens for the image.
  • Vocabularies: Text is represented using BPE-encoded tokens with a vocabulary size of 16,384. Images are compressed into a 32x32 grid of discrete latent codes using a pretrained discrete VAE, resulting in a vocabulary size of 8,192.
  • Attention Mechanism: The model features 64 self-attention layers. Image tokens can attend to all text tokens, while text tokens use a standard causal mask. Image tokens employ sparse attention patterns (row, column, or convolutional) depending on the layer.

Image Generation and Refinement

DALL·E can generate images from scratch or regenerate rectangular regions of an existing image (extending to the bottom-right corner) to maintain consistency with a text prompt. To improve output quality in demonstrations, OpenAI used CLIP to rerank the top 32 of 512 generated samples offline.

Core Capabilities

DALL·E demonstrates a diverse set of capabilities that allow it to manipulate visual concepts through language.

Compositional Control and Attributes

  • Attribute Modification: The model can modify an object's attributes and the number of times an object appears in a scene.
  • Multiple Objects: DALL·E can manage multiple objects, their attributes, and spatial relationships (e.g., "a hedgehog wearing a red hat, yellow gloves, blue shirt, and green pants"). However, performance decreases as more objects are introduced, and the model can be brittle when captions are rephrased.

Perspective and Structure

  • Viewpoint Control: DALL·E can control the scene's viewpoint and 3D rendering style, including the ability to generate smooth animations of a rotating head by drawing a figure at equally spaced angles.
  • Optical Distortions: The model can apply effects such as "fisheye lens view" and "spherical panorama."
  • Internal and External Detail: The model can render internal structures via cross-sectional views and external structures through macro photographs and "x-ray" styles.

Contextual Inference

Unlike 3D rendering engines that require unambiguous specifications, DALL·E can "fill in the blanks" for underspecified captions. For example, it can infer the need for a shadow if a caption describes a capybara in a field at sunrise, even if shadows are not explicitly mentioned.

Advanced Reasoning and Knowledge

Zero-Shot Visual Reasoning

DALL·E exhibits an emergent capability for zero-shot reasoning, allowing it to perform image-to-image translation tasks without specific training for those tasks. This capability was not explicitly encouraged during the training procedure.

World Knowledge

  • Geographic Knowledge: The model has learned facts about landmarks and neighborhoods, though its precision varies.
  • Temporal Knowledge: DALL·E can represent concepts that vary over time.

Conceptual Synthesis

  • Anthropomorphism: The model can create anthropomorphized versions of animals and objects.
  • Concept Combination: DALL·E can synthesize objects by combining disparate ideas, such as designing a product based on an unrelated concept (e.g., an armchair in the shape of an avocado).

Sources