gulucaptain/Camera-Transformer-1

What it solves

CT-1 addresses the difficulty of achieving precise, physically plausible camera control in video generation. Existing methods often rely on imprecise text prompts or labor-intensive manual trajectory parameters, making automated camera control challenging.

How it works

CT-1 uses a "Camera-Decision-First, Generation-Next" two-stage paradigm. In the first stage, the CT-1 Vision-Language-Camera (VLC) model takes an image and a text prompt describing the desired motion (e.g., "the camera moves forward and rotates left") and predicts a sequence of camera-pose encodings (the trajectory).

This process is powered by a Diffusion Transformer (DiT) combined with a novel Wavelet-based Regularization Loss in the frequency domain to learn complex trajectory distributions. These predicted trajectories are then fed into a separate video diffusion model (such as CameraCtrl or MotionCtrl) to synthesize the final video.

Who it’s for

Researchers and developers working on controllable video generation, spatial reasoning in AI, and automated cinematography for AI-generated videos.

Highlights

• VLC Model: Formulates camera trajectory estimation as a vision-language understanding task.
• Wavelet-based Regularization Loss: A novel frequency-domain loss for learning complex camera trajectories.
• CT-200K Dataset: A large-scale dataset containing over 47 million frames to support training.
• Cross-Model Compatibility: Predicted trajectories work with existing controllable video diffusion models.
• Cross-Domain Generalization: Effective across general scenes and driving scenarios.