A Dive into Text-to-Video Models

Text-to-video generation is an emerging computer vision task that creates sequences of images from text descriptions while maintaining spatial and temporal consistency. While it follows a similar evolutionary path to text-to-image generation, it is significantly more complex due to the requirement of long-term dependencies across frames.

Challenges in Text-to-Video Generation

Generating video from text is more difficult than generating static images because it requires maintaining consistency over time. The primary obstacles include:

  • Computational Costs: Ensuring temporal and spatial consistency across frames creates long-term dependencies that make training computationally expensive and often unaffordable for many researchers.
  • Data Scarcity: High-quality multi-modal datasets containing paired video and text are scarce and often sparsely annotated, which hinders the model's ability to learn complex movement semantics.
  • Captioning Ambiguity: Describing a video is more complex than describing a single image. Effective video generation often requires a sequence of prompts or a narrative story rather than a single short text prompt.

Evolution of Model Architectures

Text-to-video models have evolved through three primary architectural waves:

1. GAN and VAE-Based Approaches

Early research utilized Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) to auto-regressively generate frames based on captions. Examples include Text2Filter and TGANs-C. These early models were limited to low resolutions, short durations, and simple, isolated motions.

2. Transformer-Based Frameworks

Following the success of GPT-3 and DALL-E, researchers adopted transformer architectures to handle longer sequences. Notable models include:

  • Phenaki: Enables the generation of arbitrarily long videos conditioned on a sequence of prompts (a storyline).
  • NUWA-Infinity: Uses an autoregressive-over-autoregressive mechanism to synthesize infinite HD quality videos.
  • Other frameworks: CogVideo, VideoGPT, and Make-A-Video also utilize transformer-based approaches, while TATS combines VQGAN with a time-sensitive transformer module.

3. Diffusion-Based Architectures

The current wave of research focuses on diffusion models to achieve hyper-realistic and contextually rich results. Key developments include:

  • Video Diffusion Models (VDM): The pioneer in extending diffusion models to the video domain.
  • MagicVideo: Generates video clips in a low-dimensional latent space for increased efficiency over VDM.
  • Text2Video-Zero: A zero-shot framework that combines a trainable motion dynamics module with a pre-trained Stable Diffusion model, requiring no paired text-video data.
  • NUWA-XL: Employs a "diffusion over diffusion" method to train on 3,376 frames to reduce the context gap found in sliding-window autoregressive generation.
  • Tune-a-Video: Fine-tunes a pre-trained text-to-image model using a single text-video pair to change content while preserving motion.

Video-Text Datasets

Model performance is heavily dependent on the quality of paired video-text datasets. Current strategies include using large-scale datasets or combining different data types:

  • WebVid: A common dataset consisting of 10.7 million text-video pairs (52K video hours), though it contains significant noise and irrelevant descriptions.
  • Howto100M: Focuses on step-by-step instructional videos (cooking, gardening, etc.) with 136M clips.
  • QuerYD: Focuses on event localization, providing detailed captions regarding the relative location of objects and actions.
  • CelebV-Text: A large-scale facial dataset with over 70K videos for generating realistic faces, emotions, and gestures.

Some models, like Make-a-Video, attempt to bypass data scarcity by using text-image pairs to learn visual appearance and unimodal video data to learn spatio-temporal dependencies unsupervised.

Implementation and Open Source Resources

Many text-to-video models are now integrated into the Hugging Face ecosystem via the diffusers library. Users can run and fine-tune models such as Text2Video-Zero and ModelScope (from Alibaba / DAMO Vision Intelligence Lab).

Practical Application

Users can deploy the ModelScope model using the following implementation pattern:

import torch
from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler
from diffusers.utils import export_to_video

pipe = DiffusionPipeline.from_pretrained("damo-vilab/text-to-video-ms-1.7b", torch_dtype=torch.float16, variant="fp16")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

prompt = "Spiderman is surfing"
video_frames = pipe(prompt, num_inference_steps=25).frames
video_path = export_to_video(video_frames)

Community Ecosystem

Beyond official integrations, the community has developed unofficial implementations of several key papers (including Imagen, Phenaki, and NUWA) via contributors like Phil Wang (lucidrains), and fine-tuning frameworks like those provided by ExponentialML.

Sources