TencentARC/VerseCrafter

What it solves

VerseCrafter addresses the difficulty of maintaining geometric faithfulness and spatial-temporal coherence in AI-generated videos, specifically when trying to control the motion of the camera and multiple objects within a scene.

How it works

The system uses a frozen Wan2.1 video backbone and attaches a geometry-aware "GeoAdapter" to inject 4D geometric controls into the diffusion blocks. It utilizes a unified 4D control state to manage camera and multi-object motion. To support this, the project introduces the VerseControl4D dataset, which contains real-world videos with automatically rendered camera trajectories and 3D Gaussian trajectories for supervision.

Who it’s for

It is designed for researchers and developers working on controllable video generation and world models, as well as 3D artists who can use the provided Blender addon to visually edit trajectories.

Highlights

• 4D Geometric Control: Provides direct, interpretable control over camera and multi-object motion.
• GeoAdapter: Enables precise control without sacrificing video quality by keeping the base video prior frozen.
• Blender Integration: Includes an addon for a one-stop workflow, allowing users to visually edit trajectories in a 3D environment.
• VerseControl4D Dataset: A large-scale dataset specifically built to supervise 4D controllable video generation.