OpenBMB/VisRAG

Parsing-free RAG supported by VLMs

What it solves

VisRAG addresses the information loss that occurs when documents are parsed into text for traditional Retrieval-Augmented Generation (RAG). By treating documents as images rather than text, it preserves the original layout and visual data, maximizing the retention of information during the retrieval and generation process.

How it works

The system operates through a vision-language model (VLM) pipeline consisting of three main components:

  1. VisRAG-Ret (Retriever): A document embedding model that embeds documents directly as images to retrieve relevant visual pages based on a query.
  2. VisRAG-Gen (Generator): A VLM (such as GPT-4o or MiniCPM-V) that generates an answer based on the retrieved images.
  3. EVisRAG (VisRAG 2.0): An advanced end-to-end framework that improves multi-image reasoning. It uses a four-step process—observing images, recording evidence, reasoning, and answering—and is trained using Reward-Scoped Group Relative Policy Optimization (RS-GRPO) to optimize both visual perception and reasoning.

Who it’s for

Developers and researchers working with complex documents (like PDFs) where visual layout, charts, and images are critical to the answer and cannot be easily captured by text-only parsing.

Highlights

  • Visual-First RAG: Eliminates parsing-induced information loss by embedding documents as images.
  • Evidence-Guided Reasoning: EVisRAG uses a structured approach to collect per-image evidence before final reasoning.
  • RS-GRPO Training: Employs a specialized reinforcement learning algorithm to jointly optimize perception and reasoning.
  • Multi-Image Support: Specifically designed to handle questions that require reasoning across multiple retrieved visual documents.

Related

  • Project
    Alibaba-NLP/VRAG

    A multi-turn multimodal RAG framework that uses agentic reinforcement learning to enable VLMs to reason and retrieve information across text, images, and videos.

  • Project
    StarTrail-org/PixelRAG

    PixelRAG renders web pages, PDFs, and images into screenshot tiles, embeds them with a vision‑language model, builds a FAISS/Qdrant vector index, and provides a searchable API (or Claude plugin) so LLMs can retrieve information based on visual layout rather than plain text.

  • Project
    HKUDS/RAG-Anything

    RAG‑Anything is a Python library that builds a multimodal Retrieval‑Augmented Generation system. It parses PDFs, Office files, images, tables, and equations using MinerU, creates a cross‑modal knowledge graph, and retrieves relevant pieces with a hybrid vector‑graph search before passing them to an LLM for answer generation. Install via `pip install raganything`, configure with `RAGAnythingConfig`, ingest documents, and query in natural language.

  • Project
    BBC-Esq/VectorDB-Plugin

    A Windows-based tool for creating a vector database from documents, images, and audio files to enable Retrieval Augmented Generation (RAG) with various LLMs.

  • Project
    weiwill88/Local_Pdf_Chat_RAG

    A modular Python implementation of a RAG pipeline designed for learning and experimentation, featuring hybrid retrieval and support for multiple LLM backends.