Vchitect/Latte

What it solves

Latte is designed to generate high-quality videos from either text prompts or class labels. It addresses the challenge of modeling the complex spatio-temporal distribution of videos while managing the high computational cost associated with processing a large number of video tokens.

How it works

Latte is a Latent Diffusion Transformer. It works by first extracting spatio-temporal tokens from input videos and then using a series of Transformer blocks to model the video distribution within a latent space. To handle the large volume of tokens efficiently, it introduces four variants that decompose the spatial and temporal dimensions of the input videos. The model also incorporates specific best practices for video clip patch embedding, temporal positional embedding, and timestep-class information injection to improve output quality.

Who it’s for

This project is for AI researchers and developers working on video generation, diffusion models, and the integration of Transformers into generative video AI.

Highlights

• Multi-modal Generation: Supports both text-to-video (T2V) and text-to-image (T2I) generation.
• Efficient Architecture: Uses a Latent Diffusion Transformer with spatial-temporal decomposition to manage token counts.
• State-of-the-Art Performance: Achieves high performance across standard datasets like FaceForensics, SkyTimelapse, UCF101, and Taichi-HD.
• Broad Integration: Integrated into the Hugging Face diffusers library, supporting 4/8-bit quantization to reduce GPU memory usage.