Alibaba-NLP/VRAG

What it solves

VRAG and VimRAG address the challenge of multimodal Retrieval-Augmented Generation (RAG) across text, images, and videos. They enable Vision-Language Models (VLMs) to progressively gather information—moving from coarse-grained to fine-grained perspectives—to better understand visually rich information and navigate massive visual contexts.

How it works

The project provides two primary frameworks:

• VRAG: A purely visual RAG agent that uses iterative reasoning to gather information.
• VimRAG: A framework for multimodal reasoning that utilizes a Multimodal Memory Graph and Graph-Guided Policy Optimization (GGPO). It models the reasoning process as a dynamic directed acyclic graph (DAG), pruning redundant memory nodes to improve credit assignment and training convergence.

Both systems rely on a multimodal retrieval engine that supports embedding models like GVE and Qwen3-VL-Embedding to index and search through images, PDFs (converted to images), and video chunks.

Who it’s for

This is for AI researchers and developers building advanced multimodal agents capable of complex reasoning over large visual datasets (images, videos, and documents).

Highlights

• Multi-turn Multimodal Training: Includes a training framework (VRAG-RL) for multi-turn multimodal agent training using reinforcement learning.
• Dynamic Reasoning Visualization: VimRAG features real-time DAG visualization of the reasoning process.
• Flexible Retrieval: Supports building custom retrievers for images, PDFs, and videos using SOTA visual embedding models.
• Extensible Framework: Capable of supporting training with various tools and integrates with vLLM for local model deployment.