aigc-apps/VideoX-Fun

What it solves

VideoX-Fun provides a comprehensive pipeline for generating high-quality AI videos and images. It addresses the challenge of creating controlled video generation—allowing users to specify starting and ending frames, use reference videos for style transfer, or apply precise structural controls (like Canny edges, depth maps, or human poses) to guide the motion and composition of the generated content.

How it works

The project utilizes Diffusion Transformer (DiT) architectures, specifically supporting the CogVideoX-Fun and Wan 2.1 series models. It offers several generation modes:

• Text-to-Video (T2V): Generates video from a text prompt.
• Image-to-Video (I2V): Uses one or two images (start and end frames) to animate a scene.
• Video-to-Video (V2V): Uses a reference video to guide the generation of a new video.
• Controlled Generation: Employs control models to enforce specific conditions such as Canny, Pose, Depth, and MLSD, as well as camera movement controls (Pan Up/Down/Left/Right).

To optimize performance on consumer GPUs, it implements memory management strategies including CPU offloading and float8 quantization.

Who it’s for

This tool is designed for AI artists, content creators, and developers who want to generate cinematic AI videos with high degrees of control over motion and style. It is also suitable for researchers wanting to train their own baseline or Lora models for specific style transformations.

Highlights

• Multi-Model Support: Integrated support for CogVideoX-Fun and the Wan 2.1 series (1.3B and 14B models).
• Precise Control: Support for camera trajectory control and structural guidance (Canny, Depth, Pose).
• Custom Training: Ability to train baseline and Lora models, including Reward Lora support for human-preference optimization.
• Flexible Deployment: Available via ComfyUI, Docker, or direct Python scripts, with support for both Windows and Linux.