illuin-tech/colpali

ColPali – Visual Document Retrieval with Vision‑Language Models

What it is – ColPali is a research‑grade library that lets you turn images of document pages (PDF pages rendered as PNG/JPEG, screenshots, etc.) into multi‑vector embeddings using a Vision‑Language Model (VLM). Those embeddings can be compared to query embeddings with a late‑interaction (ColBERT‑style) similarity, enabling fast, accurate retrieval of documents that contain the information a user asks for – even when the answer is hidden in tables, charts, or layout cues.

Why it matters – Traditional document‑search pipelines first run OCR, extract layout, and then index plain text. ColPali skips that fragile OCR step: a single VLM (e.g., PaliGemma‑3B, Qwen‑VL, or Qwen3‑VL) processes the raw image patches, preserving both visual and textual cues. The result is a set of vectors per image patch that can be efficiently compared to query vectors.

Key components

• Model families – Pre‑trained checkpoints such as vidore/colpali‑v1.3, vidore/colqwen2‑v1.0, vidore/colqwen2.5‑v0.2, and community‑contributed variants (e.g., Tomoro‑colqwen3‑embed‑4b). They are built on top of the ColBERT architecture and a VLM backbone (PaliGemma, Qwen2‑VL, Qwen3‑VL, etc.).
• colpali-engine package – Provides the model classes (ColPali, ColQwen2, …), processors for images and text, and utilities for scoring, token‑pooling, and interpretability. The package can be installed from PyPI (pip install colpali-engine) or directly from source.
• Late‑interaction kernels – An optional Triton‑based late-interaction-kernels extension (colpali-engine[lik]) dramatically reduces memory use when computing the [B, B, Lq, Ld] similarity tensor, allowing larger batch sizes on modern GPUs.
• Fast‑Plaid indexing – With the plaid extra you can build a compact index (processor.create_plaid_index) that supports fast top‑k retrieval on large corpora.
• Interpretability – The interpretability extra lets you visualise similarity maps that highlight which image patches contributed most to each query token, useful for debugging and research.
• Token pooling – HierarchicalTokenPooler compresses the multi‑vector representation by merging redundant patches (e.g., white background), cutting the number of vectors by ~2/3 while keeping >97 % of retrieval performance.

Typical workflow

from colpali_engine.models import ColQwen2
from colpali_engine import ColQwen2Processor
import torch, PIL.Image as Image

model = ColQwen2.from_pretrained(
    "vidore/colqwen2-v1.0",
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",
    attn_implementation="flash_attention_2",
).eval()

processor = ColQwen2Processor.from_pretrained("vidore/colqwen2-v1.0")

# Documents (images) and queries (text)
images = [Image.open(p) for p in ["page1.png", "page2.png"]]
queries = ["What is the y‑axis variable?", "Which year had the highest outlay?"]

# Pre‑process
batch_imgs = processor.process_images(images).to(model.device)
batch_qs   = processor.process_queries(queries).to(model.device)

# Encode
with torch.no_grad():
    doc_emb = model(**batch_imgs)
    qry_emb = model(**batch_qs)

# Score (late interaction)
scores = processor.score_multi_vector(qry_emb, doc_emb)
print(scores)

The same pattern works with the newer Sentence‑Transformers v6 API (MultiVectorEncoder), which the ColPali team now recommends for production.

Current status – The original colpali-engine package is deprecated in favor of the integrated support in Sentence‑Transformers (v6+). The repository is kept for reproducibility, research, and migration guidance. All model checkpoints remain hosted on Hugging Face, and a public leaderboard (ViDoRe) tracks their retrieval performance.

Where to go next

• Production – Use sentence-transformers[image] and the MultiVectorEncoder class; it offers the same API with better ecosystem support.
• Benchmarks – The vidore-benchmark repo lets you evaluate on the ViDoRe dataset.
• Community resources – Cookbooks (github.com/tonywu71/colpali-cookbooks) contain notebooks for training, indexing, and visualising similarity maps.
• Further research – The paper (arXiv:2407.01449) and the token‑pooling work (arXiv:2409.14683) provide deeper insight into the model design.

---

TL;DR – ColPali is a library that turns document images into multi‑vector embeddings using vision‑language models, enabling fast, OCR‑free retrieval via ColBERT‑style late interaction. The original engine is now deprecated, but the models and concepts live on through Sentence‑Transformers’ multi‑vector support.