OpenBMB/VisRAG

Parsing-free RAG supported by VLMs

What it solves

VisRAG addresses the information loss that occurs in traditional text-based Retrieval-Augmented Generation (RAG) pipelines, where documents are parsed into text before retrieval. By treating documents as images and using Vision-Language Models (VLMs) for both retrieval and generation, VisRAG retains the full visual information of the original documents.

How it works

The project consists of two main iterations:

  1. VisRAG (1.0): A pipeline where documents are directly embedded as images using a VLM-based retriever (VisRAG-Ret) and then passed to a VLM generator (VisRAG-Gen) to produce an answer.
  2. EVisRAG (VisRAG 2.0): An end-to-end framework designed for multi-image reasoning. It uses a structured four-step process: observing images, recording specific evidence per image, reasoning over that evidence, and finally answering the question. It is trained using Reward-Scoped Group Relative Policy Optimization (RS-GRPO) to optimize both visual perception and reasoning.

Who it’s for

Researchers and developers building RAG systems for visually rich documents (like PDFs with charts and tables) who want to avoid the errors introduced by OCR or text parsing.

Highlights

  • Visual-First Retrieval: Embeds documents as images to maximize data retention.
  • Evidence-Guided Reasoning: EVisRAG uses a structured observation-evidence-reasoning-answer loop for multi-image tasks.
  • RS-GRPO Training: Employs a specialized reinforcement learning approach to jointly optimize perception and reasoning.
  • VLM Integration: Built upon models like Qwen2.5-VL and MiniCPM-V.