How Kapa.ai Indexes Images for RAG
Kapa.ai has implemented a strategy to integrate images into its RAG pipeline by converting images into text descriptions during the indexing phase. This approach eliminates the need to send raw images to multimodal models during every query, reducing per-query costs by 27% to 51% compared to query-time multimodal processing while statistically improving answer quality (p < 0.05).
The Role of Images in Technical Documentation
Images in technical documentation generally fall into two categories: illustrative and load-bearing. Illustrative images, such as screenshots of UI elements, clarify existing text and make instructions easier to act upon. Load-bearing images, such as wiring diagrams or specification tables, contain primary data that may not exist anywhere else in the text.
Kapa.ai found that providing image context significantly improves LLM-generated answers. In tests across three customer projects and two models, an LLM judge preferred answers with image context over text-only baselines by a statistically significant margin.
Why Query-Time Multimodal RAG Fails at Scale
Processing images during the query phase is structurally inefficient for high-volume production environments due to three primary constraints:
- Economic Costs: Raw images significantly increase token usage. In Kapa.ai's tests, images added 27% to per-query costs on GPT models and 51% on Claude models.
- Context Window Limits: A typical query retrieves 10-30 chunks, which can reference 20-30 images. With Claude's 30 MB and OpenAI's 50 MB payload limits, large numbers of images quickly exhaust the available context window.
- Retrieval Inaccuracy: CLIP-style multimodal embeddings often lack the fine-grained detail required for technical charts and tables. Furthermore, short technical queries often provide insufficient signal to match effectively against image vectors.
The Index-Time Description Strategy
To solve these issues, Kapa.ai uses an "eager processing" approach: images are described once at indexing time using a vision language model (VLM), and these captions are stored as text chunks.
At query time, the retriever pulls the relevant text caption. The LLM sees the text description and cites the original image URL without ever processing the raw pixels. For load-bearing images, the VLM transcribes the data (e.g., converting a table of ratings into a text grid), ensuring the answer remains grounded in the actual figure data.
Production Implementation Details
Image Filtering
To avoid the cost of captioning irrelevant images, Kapa.ai employs a two-step filtering process:
- Heuristics: Images are dropped based on unsupported formats, small dimensions, or extreme aspect ratios.
- Zero-Shot Classification: A classifier based on multimodal embeddings removes clear "junk" (logos, avatars, banners). This classifier achieves 96.8% accuracy on clear-cut images, though accuracy drops to 59.8% on ambiguous images where the image's purpose depends entirely on surrounding text.
Optimizing Caption Quality
Caption quality is driven more by context than by the size of the model used. Kapa.ai found that providing the VLM with the paragraphs immediately preceding and following the image significantly improved the grounding of the caption. Additionally, they found that smaller models (e.g., GPT-4o mini) produced captions nearly indistinguishable from much more expensive models, making them the most cost-effective choice for large-scale indexing.
Storage Architecture: Separate vs. Inline
Kapa.ai compared two methods of storing captions:
- Inline: Replacing image alt-text within the document, meaning every chunk containing the image also contains the caption.
- Separate: Storing each caption as its own independent chunk.
Separate chunks proved superior. Inline captions inflate the size of every chunk they inhabit, increasing costs even when the image is irrelevant. Separate chunks only enter the context window when the retriever deems them relevant. In one image-heavy project, separate chunks resulted in a 6% per-query cost increase compared to a 19% increase for inline captions.
Performance Results
Across three customer projects using GPT-4o and Claude 3.5 Sonnet, the results were as follows:
| Metric | Text-only Baseline | With Image Captions |
|---|---|---|
| Images cited in answers | 0% | 10% to 64% |
| Answer quality (LLM judge) | Baseline | Significantly better (p < 0.05) |
| Per-query cost | Baseline | +1% to 6% |
| Latency (TTFT) | Baseline | Sub-second increase |
| Model uncertainty | Baseline | Unchanged or slightly lower |
| Indexing cost | N/A | One-time cost |
Community Perspectives
While the technical approach is widely regarded as a standard pattern for media ingestion, some community members noted potential risks and benefits:
"Due to the non deterministic nature of LLMs new models will reveal new information about your data... These context adjustments might sometimes require you to rerun your LLM processing."
Other users noted that while this architecture is highly cost-efficient, it may require modifications if a query can only be answered by looking at the image in real-time, rather than relying on a pre-generated description.