StarTrail-org/PixelRAG

https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/

What it solves

Traditional Retrieval-Augmented Generation (RAG) systems often fail to process complex visual information like tables, charts, and infographics because they rely on parsing web pages or documents into raw text, which strips away essential layout and visual context. PixelRAG solves this by treating documents as visual entities, allowing AI models to "see" and retrieve information directly from screenshots.

How it works

PixelRAG replaces text-based parsing with a visual pipeline. It uses a rendering engine (via Playwright/CDP) to capture documents—such as web pages or PDFs—as high-fidelity screenshot tiles. These images are then processed by a LoRA-fine-tuned vision-language embedding model (Qwen3-VL-Embedding) that maps visual content into a vector space. This allows the system to perform semantic search over the visual structure of documents rather than just their text content.

Who it’s for

This project is for developers and researchers building RAG applications that require high accuracy for documents containing non-textual data. It is also designed for users of agentic tools like Claude Code, who want to provide their AI agents with the ability to visually interpret web pages and documents.

Highlights

  • Visual-First Retrieval: Searches documents based on how they look, preserving tables, charts, and layout structure.
  • Claude Integration: Includes a pixelbrowse plugin that allows Claude to "see" and summarize web pages by taking screenshots.
  • General-Purpose Pipeline: Provides modular tools for rendering, chunking, embedding, and indexing documents.
  • Flexible Backend: Supports local FAISS indexes for quick deployment or Qdrant for scalable, production-grade vector search.

Related

  • Project
    jolibrain/colette

    Colette is a self‑hosted RAG platform that indexes PDFs as images, preserving tables, figures and layout. It combines vision embeddings, BM25 lexical search, and optional HyDE prompting, and runs on high‑end GPUs via Docker. Ideal for confidential technical documentation.

  • Project
    HKUDS/LightRAG

    LightRAG is an open‑source Python framework that blends vector embeddings with a knowledge‑graph index for fast, scalable Retrieval‑Augmented Generation. It supports multiple chunking strategies, multimodal document parsing, incremental updates, and a variety of storage back‑ends, and ships with a REST API and Web UI.

  • Project
    ggozad/haiku.rag

    haiku.rag is a Python library/CLI for building a private, citation‑rich RAG system over PDFs, Office files, HTML, Markdown and images. It uses LanceDB for hybrid vector/full‑text indexing, Docling for document parsing, and Pydantic AI for LLM/embedding providers (defaulting to Ollama). Features include hybrid search, optional reranking, multimodal image search, sandboxed Python for data aggregation, and a CLI/TUI that shows exact page‑level citations. It offers an optional background ingester service and an MCP server for integration with tools like Claude Code. Install via `pip install haiku.rag` (or the slim core) and interact via commands like `haiku-rag add-src`, `haiku-rag ask`, or the async Python API.

  • Project
    Marker-Inc-Korea/AutoRAG

    AutoRAG Agent is a Node‑based, locally‑run RAG system that turns your computer into a searchable knowledge base. It indexes files (PDF, Office, HWP, images, etc.) with BM25 and a built‑in embedding model, supports many external chat sources, and answers natural‑language questions with numbered citations. Available as a CLI, an interactive terminal UI, a desktop Electron app, and a TypeScript library, it emphasizes privacy (no data leaves the machine) and ease of use (no vector‑DB setup).

  • Project
    OpenRaiser/PaperFit

    PaperFit is an AI‑agent system that automatically checks and repairs the visual layout of LaTeX papers. It renders PDFs to images, diagnoses layout defects, proposes LaTeX edits, and iterates until the paper meets visual and page‑budget goals. It works with Claude Code, Codex and Cursor, and supports template migration, page‑count control, and local object fixes.