gudaochangsheng/RefAlign

[ECCV 2026] Official PyTorch implementation of RefAlign: Representation Alignment for Reference-to-Video Generation

What it solves

Reference-to-video (R2V) generation often struggles with "copy-paste artifacts" and "multi-subject confusion," where the model fails to maintain the identity of the reference subject or confuses multiple subjects in a scene. RefAlign addresses these issues by improving reference fidelity, identity consistency, and text controllability.

How it works

RefAlign is a training-time representation alignment framework. It explicitly aligns the features of a Diffusion Transformer (DiT) reference branch to the feature space of a frozen visual foundation model (VFM), such as DINOv3. It uses a reference alignment loss that pulls features of the same subject closer together and pushes different subjects further apart. Because this alignment happens only during training, the VFM and alignment module are discarded during inference, resulting in no additional computational overhead.

Who it’s for

This project is for AI researchers and developers working on controllable video synthesis, personalized advertising, virtual try-on, and identity-consistent video creation.

Highlights

  • Zero Inference Overhead: The alignment process is only used during training, so it doesn't slow down video generation.
  • SOTA Performance: Achieves state-of-the-art results on the OpenS2V-Eval benchmark.
  • Flexible Model Sizes: Provides checkpoints for both 1.3B and 14B parameter models.
  • Improved Fidelity: Specifically targets and reduces common R2V artifacts like copy-paste effects.

Related

  • Project
    X-GenGroup/Flow-Factory

    A unified framework for online RL and offline fine-tuning of diffusion and flow-matching models across image, video, and audio-video modalities.

  • Project
    WeChatCV/Stand-In

    A lightweight, plug-and-play framework for identity-preserving video generation that maintains high subject consistency by training only 1% of additional parameters.

  • Project
    microsoft/World-R1

    World-R1 is a reinforcement learning framework that improves 3D geometric consistency in text-to-video generation by aligning generated videos with 3D constraints without changing the base model architecture.

  • Dispatch
    Google DeepMind Visual Alignment Research

    Google DeepMind has developed a three-step alignment method that reorganizes AI visual representations to match human conceptual hierarchies, improving model robustness and generalization.

  • Project
    ReinFlow/ReinFlow

    A policy gradient framework for fine-tuning flow matching policies using online reinforcement learning, enabling efficient optimization of robotic control policies with minimal denoising steps.