tonywu71/colpali-cookbooks

Recipes for learning, fine-tuning, and adapting ColPali to your multimodal RAG use cases. 👨🏻‍🍳

What it solves

This project provides practical implementation guides for using the ColPali family of Vision Language Models (VLMs) for document retrieval. It addresses the difficulty of traditional retrieval systems that rely on OCR and text extraction, which often lose critical visual information like layout, tables, and charts.

How it works

The project uses models like ColPali and ColQwen2, which treat document pages as images rather than text. These models use a vision-language approach to capture visual features and create multi-vector embeddings. Retrieval is then performed by computing pairwise late interaction similarity scores between the query and the document image embeddings.

Who it’s for

Developers and researchers interested in implementing vision-based document retrieval, building RAG pipelines with VLMs, and those looking to fine-tune these models for specific use cases.

Highlights

  • End-to-End RAG: Implementation of a full RAG pipeline using ColQwen2 with adapter hot-swapping to save VRAM.
  • Fine-tuning: Guides on fine-tuning ColPali using LoRA and optional 4-bit or 8-bit quantization.
  • Interpretability: Tools to generate similarity maps to visualize and interpret model predictions.
  • Native Integration: Examples for using the transformers-native implementations of ColPali and ColQwen2.

Related

  • Project
    illuin-tech/colpali

    ColPali is a research library for visual document retrieval. It encodes document images with vision‑language models (e.g., PaliGemma, Qwen‑VL) into multi‑vector embeddings and scores them against query embeddings using a ColBERT‑style late‑interaction method. The original `colpali-engine` package (now deprecated) provides model classes, processors, optional fused kernels, fast‑Plaid indexing, token‑pooling, and interpretability tools. New projects should use Sentence‑Transformers v6’s `MultiVectorEncoder`, which integrates ColPali‑style models.

  • Project
    OpenBMB/VisRAG

    A vision-language RAG pipeline that embeds documents as images to prevent information loss from text parsing and uses evidence-guided reasoning for multi-image QA.

  • Project
    morphik-org/morphik-core

    A multimodal retrieval engine that enables AI applications to search and understand visually rich documents, such as PDFs and charts, without losing spatial or visual context.

  • Project
    gokayfem/ComfyUI_VLM_nodes

    A production-oriented suite of ComfyUI nodes for vision-language models, providing structured detection, segmentation, and adaptive video reasoning with VRAM optimizations.

  • Project
    vespa-engine/sample-apps

    A collection of reference applications and tutorials for Vespa, demonstrating how to implement hybrid search, RAG, and multimodal vector retrieval at scale.