PKU-YuanGroup/ConsisID

What it solves

ConsisID addresses the challenge of maintaining consistent human identity across frames in text-to-video generation. It prevents the "shifting" of a person's appearance that often occurs in AI-generated videos, ensuring the subject looks like the same individual throughout the clip.

How it works

ConsisID is a tuning-free, Diffusion Transformer (DiT)-based model. It utilizes frequency decomposition, drawing on frequency analysis of vision and diffusion transformers, to preserve identity. The system takes a reference image of a person and a text prompt to generate a video where the identity from the image is consistently applied to the subject in the video.

Who it’s for

This tool is designed for creators and researchers working with controllable video generation who need high-fidelity identity preservation without the need for extensive model fine-tuning.

Highlights

• Tuning-Free: Maintains identity consistency without requiring additional training or fine-tuning for new subjects.
• DiT-Based: Built on the Diffusion Transformer architecture.
• Memory Optimizations: Includes options like CPU offloading, VAE slicing, and tiling to allow the model to run on GPUs with less than 44GB of VRAM.
• Integration: Integrated into the Hugging Face diffusers library.
• Inference Acceleration: Supports parallel inference via xDiT and cached inference via TeaCache.