Weistrass/DyRef

[ECCV 2026 Oral] Official repository for Scaling Multi-Reference Image Generation with Dynamic Reward Optimization

What it solves

DyRef addresses the performance drop seen in open-source image generation models when they are asked to handle multiple reference images of different types (e.g., combining a specific subject, a specific style, and a specific pose) simultaneously. It prevents the model from ignoring some references or failing to combine them accurately as the complexity of the request increases.

How it works

DyRef uses a two-stage training framework to improve multi-reference image generation (MRIG):

  1. Stage I (SFT): The model is first trained via Supervised Fine-Tuning (SFT) to establish basic capabilities for handling complex multi-reference tasks.
  2. Stage II (RL): The model undergoes Reinforcement Learning (RL) using two specific mechanisms:
    • Dynamic Reward Shaping (DRS): Increases the reward differences between samples to make training more effective.
    • Dynamic Attention Reward (DAR): Forces the model to focus more on samples that contain a high number of mixed-type reference images.

Additionally, the project provides an automated data synthesis pipeline to create the training data needed for these complex tasks.

Who it’s for

Researchers and developers working on controllable image generation, creative AI tools, and video keyframe generation where precise control over multiple visual attributes is required.

Highlights

  • OmniRef-Bench: A new benchmark to evaluate how well models preserve and combine diverse reference types (subject, style, background, lighting, and pose).
  • Broad Compatibility: Enhances multiple foundation models, including Qwen-Image-Edit-2511 and FLUX.2-klein-base.
  • Performance Gains: Achieves results comparable to high-end closed-source models like Nano Banana Pro.
  • No Regression: Improves multi-reference capabilities without compromising the general capabilities of the base foundation models.

Related

  • Project
    X-GenGroup/Flow-Factory

    A unified framework for online RL and offline fine-tuning of diffusion and flow-matching models across image, video, and audio-video modalities.

  • Project
    Alibaba-NLP/VRAG

    A multi-turn multimodal RAG framework that uses agentic reinforcement learning to enable VLMs to reason and retrieve information across text, images, and videos.

  • Project
    Tencent-Hunyuan/UniRL

    UniRL is an open‑source Python framework that applies a unified reinforcement‑learning loop to many multimodal generative models (LLMs, vision‑language models, image/video diffusion, and hybrid AR‑diffusion models). It provides four entry points, Hydra‑based configs, pluggable rollout engines, and distributed training via Ray and FSDP. The repo ships standard RL algorithms plus three novel team‑proposed methods (Flow‑DPPO, DRPO, CPPO) with tutorials. Supported models include Stable Diffusion 3, FLUX.2‑Klein, Qwen‑3, HunyuanVideo, and more. Documentation, example recipes, and a WeChat community are provided. Licensed under Apache‑2.0.

  • Project
    SkyworkAI/UniPic

    A unified multimodal model series for image editing, generation, and understanding, featuring high-speed multi-image editing and autoregressive modeling.

  • Project
    deepgenteam/deepgen

    DeepGen 1.0 is a lightweight 5B parameter unified multimodal model that integrates image generation, editing, and reasoning into a single framework.