Tencent-Hunyuan/HunyuanVideo-I2V
HunyuanVideo-I2V: A Customizable Image-to-Video Model based on HunyuanVideo
What it solves
HunyuanVideo-I2V is an image-to-video generation framework that allows users to create high-quality videos (up to 720p and 129 frames) starting from a single reference image and a text prompt. It specifically addresses the challenge of maintaining visual consistency between the reference image and the first frame of the generated video while allowing for customizable effects via LoRA training.
How it works
The system extends the HunyuanVideo architecture by using a token replace technique to integrate reference image information. It utilizes a pre-trained Multimodal Large Language Model (MLLM) with a Decoder-Only architecture as a text encoder. The MLLM processes the input image into semantic image tokens, which are then concatenated with video latent tokens. This allows the model to perform full-attention computation across both image and text modalities, ensuring the generated video is semantically aligned with the input image and caption.
Who it’s for
This tool is designed for AI researchers, developers, and creators who want to generate cinematic videos from static images, as well as those looking to train custom video effects using LoRA.
Highlights
- High Resolution: Supports video generation up to 720p resolution.
- Visual Consistency: Optimized to ensure the first frame of the video remains faithful to the reference image.
- Customizable Effects: Includes LoRA training scripts to create specific video effects (e.g., hair growth or embracing).
- Parallel Inference: Supports multi-GPU sequence parallel inference via xDiT for faster generation speeds.
- Flexible Dynamics: Offers configurations to toggle between stable video generation and high-dynamic movement.