StarTrail-org/PixelRAG

https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/

PixelRAG – Retrieval‑Augmented Generation on Screenshots

PixelRAG is an open‑source library that lets you search and retrieve information from documents by the way they look, not just by their raw text. It renders web pages, PDFs, or images into tiled screenshots, embeds those images with a vision‑language model, and builds a vector index that can be queried with text or images.

What it does

  • Render any URL, PDF, or image into a set of high‑resolution screenshot tiles (pixelshot).
  • Embed the tiles with a LoRA‑fine‑tuned Qwen3‑VL‑Embedding model that captures visual layout (tables, charts, infographics).
  • Index the embeddings with FAISS (or Qdrant) to enable fast similarity search.
  • Serve a REST API (pixelrag serve) that accepts text or image queries and returns the most relevant screenshot tiles.
  • Provides a Claude Code plugin (pixelbrowse) so Claude can “see” a page by calling pixelshot locally.

Core components

Command Role Install extra
pixelshot Convert a document → image tiles (uses Playwright/Chromium) pip install pixelrag
pixelrag chunk / embed / build-index Turn tiles → vectors → FAISS index pip install 'pixelrag[embed]'
pixelrag index Orchestrates full pipeline (source → ingest → embed → index) pip install 'pixelrag[index]'
pixelrag serve Launch a FastAPI search service (CPU or GPU) pip install 'pixelrag[serve]'
Qdrant backend Optional disk‑backed, scalable vector store pip install 'pixelrag[serve,qdrant]'

Quick start (no setup needed)

# Render a page to tiles
pixelshot https://en.wikipedia.org/wiki/Python -o ./tiles

# Query the public Wikipedia index (8.28 M pages)
curl -X POST https://api.pixelrag.ai/search \
  -H "Content-Type: application/json" \
  -d '{"queries":[{"text":"What is the capital of France?"}],"n_docs":5}'

The hosted API returns the screenshot tiles that contain the answer, letting any downstream LLM read the visual content directly.

Building your own index

# pixelrag.yaml
source:
  type: local
  path: ./my_docs
embed:
  model: Qwen/Qwen3-VL-Embedding-2B
  device: auto   # picks CUDA, MPS, or CPU
output: ./my_index

pip install 'pixelrag[index]'
pixelrag index build          # renders → embed → FAISS index
pixelrag serve --index-dir ./my_index --port 30001

You can then query the local server with the same curl request shown above.

Claude “eyes” plugin

  1. Install the CLI so pixelshot is on your PATH (uv tool install pixelrag or pipx install pixelrag).
  2. Add the plugin in Claude Code Marketplace and install pixelbrowse@pixelrag-plugins.
  3. Use commands like:
    claude -p "screenshot https://news.ycombinator.com and summarize the top stories"
    Claude will call pixelshot, receive the screenshot tiles, and answer based on the visual layout.

Training (optional)

The repo includes a separate train/ folder that fine‑tunes the embedding model on a large screenshot dataset (LoRA on Qwen3‑VL‑Embedding‑2B). Pre‑trained adapters are published on Hugging Face, so you can use them out‑of‑the‑box without training.

Where to find more

  • Live demo & docs: https://pixelrag.ai
  • API reference: https://pixelrag.ai/docs
  • Colab quickstart notebook: linked in the README
  • Pre‑built Wikipedia index: downloadable from the Hugging Face dataset StarTrail-org/pixelrag-faiss-indexes (≈217 GB).

License

Apache‑2.0 – free for commercial and research use.

PixelRAG turns visual structure into a searchable knowledge base, enabling LLMs (Claude, GPT, etc.) to answer questions that depend on tables, charts, and layout that traditional text‑only RAG would miss.

Related

  • Project
    DocAILab/XRAG

    XRAG is an open‑source Python framework that benchmarks the individual components of Retrieval‑Augmented Generation (RAG) pipelines. It offers modular plug‑ins for retrievers, embeddings, LLMs, and orchestrators, supports traditional IR metrics plus LLM‑based quality scores, provides a CLI (`xrag-cli`) and a Web UI, and can run as a REST service. Install via PyPI, configure with `config.toml`, and use commands like `xrag-cli run`, `xrag-cli webui`, or `xrag-cli api` to evaluate or serve RAG systems.

  • Project
    jolibrain/colette

    Colette is a self‑hosted RAG platform that indexes PDFs as images, preserving tables, figures and layout. It combines vision embeddings, BM25 lexical search, and optional HyDE prompting, and runs on high‑end GPUs via Docker. Ideal for confidential technical documentation.

  • Project
    RUC-NLPIR/FlashRAG

    FlashRAG is a Python library that provides a modular framework, benchmark datasets, and implementations of 23 SOTA Retrieval‑Augmented Generation (RAG) methods (including reasoning‑based and multimodal approaches). It supports dense, sparse, and neural‑sparse retrieval, fast LLM inference via vLLM, and offers a web UI for easy configuration and evaluation.

  • Project
    FalkorDB/GraphRAG-SDK

    GraphRAG‑SDK is a Python SDK that builds a knowledge graph in FalkorDB from your documents and then answers questions by traversing that graph. It adds provenance edges, multi‑hop retrieval, and optional abstention to dramatically cut down LLM hallucinations, and it ships with benchmarks showing it outperforms standard vector‑RAG. Install via pip, run a FalkorDB container, ingest texts (or PDFs/markdown), call `finalize()`, and query with `completion()`. The library supports schemas, incremental updates, custom providers, and is Apache‑2.0 licensed.

  • Dispatch
    Retrieval Augmented Generation with Hugging Face Transformers and Ray

    Hugging Face has integrated Ray into the Retrieval Augmented Generation (RAG) model's document retrieval mechanism to achieve a 2x speedup in retrieval calls and improve distributed fine-tuning scalability.