aigc-apps/VideoX-Fun

What it solves

VideoX-Fun provides a comprehensive pipeline for generating high-quality AI images and videos. It addresses the complexity of deploying and training Diffusion Transformer (DiT) models by offering a unified framework for inference and training across multiple model families, including CogVideoX and Wan2.1/2.2.

How it works

The project implements a versatile generation pipeline that supports several modalities:

• Inference: Users can generate content via Python scripts, a Gradio-based WebUI, or ComfyUI. It supports text-to-video (T2V), image-to-video (I2V), video-to-video (V2V), and controlled video generation (using Canny, Pose, Depth, etc.).
• Memory Optimization: To enable usage on consumer GPUs, it includes GPU_memory_mode options such as model_cpu_offload, model_cpu_offload_and_qfloat8 (quantization), and sequential_cpu_offload.
• Training: It provides a full pipeline for data preprocessing (segmentation, cleaning, and captioning) and training both baseline models and LoRA (Low-Rank Adaptation) models for specific style transformations.
• Parallelization: For large models, it supports multi-GPU inference using xfuser and yunchang to distribute workloads across heads and sequences.

Who it’s for

• AI Artists and Creators: Who want to generate videos from text, images, or existing videos with precise control.
• ML Researchers and Developers: Who need a standardized pipeline to train or fine-tune Diffusion Transformer models using their own datasets.
• GPU-Constrained Users: Those looking to run large video models on consumer-grade hardware via quantization and offloading.

Highlights

• Multi-Model Support: Compatible with CogVideoX-Fun, Wan2.1, Wan2.2, and Qwen-Image series.
• Flexible Control: Supports advanced control mechanisms including Canny, Pose, Depth, MLSD, trajectory control, and camera lens control.
• Sizable Training Suite: Includes scripts for baseline training, LoRA training, and reward-based LoRA.
• Deployment Options: Offers multiple entry points including a WebUI, ComfyUI integration, and direct Python API.