OpenAI Sora: Video Generation Models as World Simulators

OpenAI has introduced Sora, a diffusion transformer model capable of generating high-fidelity video up to one minute long. This development suggests that scaling video generation models is a promising path toward building general-purpose simulators of the physical world.

Unified Visual Representation via Spacetime Patches

Sora utilizes a unified representation of visual data called "patches," which allows the model to train on videos and images of varying durations, resolutions, and aspect ratios. This approach is inspired by the tokenization used in Large Language Models (LLMs) to unify diverse text modalities.

Video Compression Network

To enable large-scale training, OpenAI uses a network to reduce the dimensionality of raw video, outputting a latent representation that is compressed both spatially and temporally. A corresponding decoder model is then used to map these generated latents back into pixel space.

Spacetime Latent Patches

Once compressed, the video is decomposed into a sequence of spacetime patches that serve as transformer tokens. Because images are treated as videos with a single frame, this same patch-based scheme applies to both. At inference time, the size of the generated video is controlled by arranging randomly initialized patches in a grid of the desired size.

Scaling Diffusion Transformers for Video

Sora is a diffusion transformer. It is trained to predict the original "clean" patches from input noisy patches, conditioned on information such as text prompts. OpenAI found that diffusion transformers scale effectively for video generation, with sample quality improving markedly as training compute increases.

Native Aspect Ratio Training

Unlike previous models that cropped or resized data to a standard size (e.g., 256x256), Sora is trained on data at its native size. This provides two primary benefits:

• Sampling Flexibility: Sora can generate widescreen (1920x1080p), vertical (1080x1920), and intermediate aspect ratios natively.
• Improved Composition: Training on native aspect ratios reduces the likelihood of subjects being partially cut out of the frame, leading to better overall framing and composition.

Language Understanding and Prompting

To improve text fidelity and video quality, OpenAI applied the re-captioning technique from DALL·E 3. A highly descriptive captioner model was trained to produce detailed text captions for the training set. Additionally, GPT is used to expand short user prompts into the longer, detailed captions required by the video model.

Multimodal Prompting

Beyond text-to-video, Sora supports several other input modalities:

• Image-to-Video: Sora can animate static images (including those from DALL·E 2 and 3) using an image and a text prompt.
• Video Extension: The model can extend existing videos both forward and backward in time, enabling the creation of seamless infinite loops.
• Video-to-Video Editing: Using techniques like SDEdit, Sora can transform the styles and environments of input videos in a zero-shot manner.
• Video Interpolation: Sora can create seamless transitions between two different input videos by gradually interpolating between them.
• Image Generation: By arranging patches in a spatial grid with a temporal extent of one frame, Sora can generate images up to 2048x2048 resolution.

Emergent Simulation Capabilities

Scaling the model has led to the emergence of capabilities that allow Sora to simulate aspects of the physical and digital world without explicit inductive biases for 3D or objects.

• 3D Consistency: Sora maintains consistent movement of people and scene elements through 3D space even during dynamic camera motion.
• Long-range Coherence: The model can persist objects, animals, and people even when they are occluded or leave the frame, and can maintain character appearance across multiple shots in a single sample.
• Physical Interaction: Sora can simulate simple state changes, such as a painter leaving persistent strokes on a canvas or a person leaving bite marks in a burger.
• Digital World Simulation: Sora can simulate artificial processes, such as rendering the world and dynamics of Minecraft in high fidelity when prompted.

Current Limitations

Despite these emergent capabilities, Sora is not yet a perfect simulator. It struggles with the physics of complex interactions, such as the shattering of glass, and does not always produce correct changes in object state when eating food. Additionally, some long-duration samples exhibit incoherencies or the spontaneous appearance of objects.