jolibrain/colette

Colette – Self‑hosted Vision‑RAG platform

What it is – Colette is an open‑source, self‑hosted Retrieval‑Augmented Generation (RAG) system that lets you index and query technical documents without sending any data to external APIs. Its distinguishing feature is Vision‑RAG: every page of a document is treated as an image, so tables, diagrams, schematics and layout are kept intact and can be searched with vision‑based embeddings.

Key capabilities

• Vision‑RAG (V‑RAG) – uses vision embedding models (e.g. gme‑Qwen2‑VL) and multimodal LLMs (e.g. Qwen3‑VL) to embed page‑level images and retrieve them.
• Hybrid retrieval – runs a vector‑search engine (ChromaDB/ColDB) and a BM25 lexical engine (Tantivy) in parallel, which is useful for product codes, part numbers, etc.
• HyDE (Hypothetical Document Embeddings) – optionally generates a short “hypothetical answer” with the LLM, embeds that instead of the raw question, and improves recall on technical queries.
• Text‑only RAG – classic pipeline using unstructured + langchain for documents where image‑based processing isn’t needed.
• Multi‑backend LLM support – works with HuggingFace, vLLM, Ollama, and a custom vllm_client.
• Image generation – integrates Stable Diffusion via diffusers for visual output.
• Layout detection – automatically crops pages into text, figure, table, or full_page regions, enabling fine‑grained filtering of results.
• Dockerised deployment – ready‑to‑run images (colette_gpu, colette_gpu_server, colette_ui) and a CI pipeline for building/publishing them.

Architecture snapshot – The README includes a diagram showing three main components: (1) Ingestion (PDF → image crops → vision/text embeddings), (2) Storage (vector DB + BM25 index), and (3) Inference (LLM generates answers using retrieved context). All components are configurable via JSON files.

Typical workflow

1. Install – Docker (recommended) or a Python virtual environment. Requires Python ≥ 3.12, CUDA ≥ 12.1, a GPU with ≥ 24 GB VRAM (or lighter models via config).
2. Index – colette_cli index … points to a folder of PDFs; the system creates image crops, runs the vision/text encoders, and stores embeddings.
3. Query – colette_cli chat … or the Python API (JSONApi, APIData) sends a question; the system retrieves relevant image/text chunks (embedding, BM25, or hybrid) and feeds them to the LLM, returning a textual answer plus source images.

Technology stack

• Languages: Python (core logic, CLI, API) and Bash (Docker wrappers).
• Vision models: Qwen‑style multimodal encoders.
• LLM back‑ends: HuggingFace Transformers, vLLM, Ollama.
• Vector stores: ChromaDB or custom coldb.
• Lexical search: Tantivy (Rust‑based BM25).
• Containerisation: Docker + Docker‑Compose; CI via Jenkins.
• Optional GPU‑accelerated image generation: Diffusers (Stable Diffusion).

Ease of setup – Moderate to high. The Docker image abstracts most dependencies, but the default configuration expects a high‑end GPU. The repo provides helper scripts for virtual‑env setup on x86, ARM, and DGX machines, plus detailed CLI examples and Jupyter notebooks for debugging.

Who would use it

• Companies or research groups that need to query proprietary technical manuals, aerospace documentation, schematics, or any PDF where visual layout matters and data cannot leave the premises.
• Teams building internal knowledge‑base assistants that must respect strict confidentiality.
• Developers experimenting with multimodal RAG pipelines and wanting a ready‑made, configurable stack.

Maturity – The project is backed by Jolibrain, CNES, and Airbus, includes a CI pipeline, Docker images, and a fairly extensive documentation site. It appears production‑ready for self‑hosted use, though the hardware requirements (≥ 24 GB VRAM) limit it to workstation‑class or server‑grade GPUs.

---

Quick start (Docker)

# pull image
docker pull docker.jolibrain.com/colette_gpu:latest

# create folders
mkdir -p models app_colette docs/pdf

# index your PDFs
docker run --gpus all -v $(pwd):/rag -v $(pwd)/docs/pdf:/data -v $(pwd)/models:/app/models \
  docker.jolibrain.com/colette_gpu \
  bash -c "colette_cli index --app-dir /rag/app_colette --data-dir /data --config-file src/colette/config/vrag_default_lite.json --models-dir /app/models"

# ask a question
docker run --gpus all -v $(pwd):/rag -v $(pwd)/app_colette:/app/app_colette -v $(pwd)/models:/models \
  docker.jolibrain.com/colette_gpu \
  bash -c "colette_cli chat --app-dir app_colette --models-dir /models --msg \"What are the identified sources of errors of a RAG?\""

Resources

• Demo video: https://github.com/user-attachments/assets/7e36b4af-880a-4260-af61-3041b7d60439
• Full docs: https://www.colette.chat/doc/
• FAQ & issue tracker: https://github.com/jolibrain/colette/issues