alibaba/Tora

What it solves

Tora addresses the challenge of generating high-quality videos with controllable motion. While standard Diffusion Transformers (DiT) can produce visually impressive content, they often lack precise control over how objects move within a scene. Tora allows users to guide video generation using specific trajectories, ensuring that the motion of the content follows a defined path.

How it works

Tora is a trajectory-oriented Diffusion Transformer framework that combines textual, visual, and trajectory conditions. It uses three primary components:

1. Trajectory Extractor (TE): Encodes arbitrary motion trajectories into hierarchical spacetime motion patches using a 3D video compression network.
2. Spatial-Temporal DiT: The core transformer architecture that generates the video frames.
3. Motion-guidance Fuser (MGF): Integrates the motion patches into the DiT blocks, ensuring the generated video consistently follows the provided trajectories.

Who it’s for

This project is designed for AI researchers and developers working on video generation, specifically those who need precise control over object dynamics and physical movement in synthetic videos.

Highlights

• Multi-Condition Guidance: Simultaneously integrates text, images, and trajectory paths for generation.
• T2V and I2V Support: Supports both Text-to-Video and Image-to-Video workflows.
• Scalable Design: Aligns with DiT scalability to support various resolutions, aspect ratios, and durations.
• High Motion Fidelity: Specifically designed to simulate the movement of the physical world with high precision.