PKU-YuanGroup/MagicTime

What it solves

MagicTime addresses the difficulty of generating "metamorphic videos"—time-lapse sequences that depict significant physical changes over time, such as plants growing, ice melting, or buildings being constructed. These types of videos are harder to generate than general videos because they require the model to understand physical knowledge, maintain long-term persistence of objects, and handle strong visual variations.

How it works

MagicTime is a metamorphic video generation pipeline that uses a specialized dataset and proposed methods to enhance the capacity of video generation models to accurately depict real-world transformations. It utilizes the ChronoMagic dataset, which contains thousands of time-lapse video-text pairs captioned by models like GPT-4V and GPT-4o, to train the model to simulate these metamorphic processes based on text prompts.

Who it’s for

This project is for AI researchers and developers working on text-to-video generation, specifically those interested in simulating physical transformations and time-lapse effects.

Highlights

• Specialized in Metamorphic Generation: Specifically designed for time-lapse videos showing growth, decay, or construction.
• ChronoMagic Dataset: Includes multiple versions of a time-lapse dataset, ranging from a small subset of 2,265 pairs to a professional version with 460K pairs.
• Physical Realism: Aims to better capture real-world physical knowledge and long-term persistence compared to general video models.