caiyuanhao1998/Open-OmniVCus

What it solves

OmniVCus addresses the challenge of subject-driven video customization, allowing users to generate videos featuring a specific subject (defined by a reference image) while maintaining precise control over the video's content through various multimodal conditions.

How it works

The project utilizes a diffusion Transformer (DiT) architecture and a specialized data construction pipeline. It enables feedforward subject customization by integrating multimodal control conditions—such as depth maps, masks, motion (optical flow), and camera poses—into the generation process. The implementation leverages Lottery Embedding (LE) and Temporally Aligned Embedding (TAE) to handle these control signals effectively. It is built upon base models like Wan2.1 and Wan2.2.

Who it’s for

This tool is designed for researchers and developers in the field of AI video generation who want to create videos with consistent subjects under strict structural or motion constraints.

Highlights

• Multimodal Control: Supports a variety of control conditions including depth, masks, and motion.
• Subject-Driven: Uses a reference image to ensure the subject remains consistent across the generated video.
• Comprehensive Pipeline: Includes a data construction factory (VideoCus-Factory) to create the necessary multimodal pairs from raw videos.
• Flexible Model Sizes: Provides implementations based on different model scales, including 1.3B and 14B parameter versions.