gudaochangsheng/RefAlign

What it solves

Reference-to-video (R2V) generation often struggles with "copy-paste artifacts" and "multi-subject confusion," where the generated video fails to maintain the identity of the reference image or confuses multiple subjects. RefAlign addresses these issues by improving reference fidelity, identity consistency, and text controllability.

How it works

RefAlign is a training-time alignment framework. It explicitly aligns the features of a Diffusion Transformer (DiT) reference branch to the feature space of a frozen visual foundation model (VFM), such as DINOv3. It uses a reference alignment loss that pulls features of the same subject closer together and pushes features of different subjects apart. Because this alignment happens only during training, the VFM and the alignment module are discarded at inference time, meaning there is no additional computational overhead during video generation.

Who it’s for

This project is designed for researchers and developers working on controllable video synthesis, personalized advertising, virtual try-on, and identity-consistent video creation.

Highlights

• State-of-the-art performance: Achieves top scores on the OpenS2V-Eval benchmark.
• Zero inference overhead: The alignment process is restricted to training, leaving the inference speed unaffected.
• Multiple model sizes: Provides both 1.3B and 14B parameter checkpoints.
• Improved consistency: Specifically targets and reduces artifacts and subject confusion in reference-guided video generation.