Anthropic Contextual Retrieval technique boosts RAG accuracy

TL;DR

Anthropic introduced Contextual Retrieval, a technique that prepends concise, chunk‑specific context to each document chunk before creating embeddings and BM25 indexes, reducing top‑20‑chunk retrieval failures by 49% with contextual embeddings alone and by 67% when combined with reranking. The method is deployable today via Claude and a public cookbook, offering a cost‑effective way to improve Retrieval‑Augmented Generation (RAG) for large knowledge bases.

Why traditional RAG loses context

Traditional RAG splits a corpus into small chunks, embeds each chunk, and searches a vector database for semantic similarity. This approach often discards the surrounding document context, leading to ambiguous chunks that lack essential identifiers (e.g., a revenue‑growth sentence without the company name). The loss of context can cause the system to retrieve irrelevant or incomplete information, degrading downstream model performance.

Core idea: Contextual Retrieval

Contextual Retrieval solves the context loss problem by prepending explanatory metadata to each chunk before embedding and before building the BM25 index. The added metadata ("contextualized chunk") describes the chunk’s provenance—document title, section, date, or any other salient details—while remaining short (≈50‑100 tokens).

Example transformation

original_chunk = "The company's revenue grew by 3% over the previous quarter."

contextualized_chunk = "This chunk is from an SEC filing on ACME Corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter."

The contextualized chunk is then embedded (Contextual Embeddings) and indexed with BM25 (Contextual BM25).

Implementation pipeline

Chunk the corpus – split documents into ≤ few‑hundred‑token pieces.
Generate contextual metadata – use Claude 3 Haiku with a prompt that receives the whole document and a target chunk and returns a concise context string.
Prepend context – attach the generated metadata to the original chunk.
Create embeddings – feed the contextualized chunk to an embedding model (e.g., Gemini Text‑004, Voyage).
Build BM25 index – index the same contextualized text with TF‑IDF‑based BM25.
Runtime retrieval – query the vector store and BM25, fuse results, deduplicate, and pass the top‑K chunks (typically 20) to the generative model.
Optional reranking – run a reranker (e.g., Cohere) on the initial top‑N (≈150) chunks, then keep the top‑K for final prompting.

A visual overview of steps 2‑5 is provided in Anthropic’s blog image.

Quantitative impact

Anthropic evaluated Contextual Retrieval across multiple domains (code, fiction, ArXiv, scientific papers) and embedding providers. The primary metric was 1 – recall@20 (percentage of relevant chunks missed in the top‑20). Results:

Configuration	Failure rate (1 – recall@20)	Relative improvement
Baseline embeddings only	5.7 %	–
+ Contextual Embeddings	3.7 %	35 % reduction
+ Contextual Embeddings + Contextual BM25	2.9 %	49 % reduction
+ Reranking (Cohere) on top‑150 → top‑20	1.9 %	67 % reduction

All tested embedding models benefitted, with Gemini and Voyage showing the strongest absolute gains.

Cost‑effective deployment with Claude prompt caching

Claude’s prompt caching lets developers load a whole document into a cache once and reuse it for every chunk’s contextualization step. Assuming 800‑token chunks, 8 k‑token documents, a 50‑token context‑generation prompt, and ~100‑token context per chunk, the one‑time cost is $1.02 per million document tokens. This makes large‑scale contextualization affordable.

Practical considerations

Chunk boundaries – Choose size, overlap, and breakpoints that preserve logical units; overly small chunks may dilute context, while overly large chunks increase latency.
Embedding model selection – While Contextual Retrieval improves all models, Gemini Text‑004 and Voyage embeddings yielded the best absolute performance in Anthropic’s tests.
Custom prompts – Tailoring the Claude prompt (e.g., adding a domain‑specific glossary) can further boost contextual relevance.
Number of retrieved chunks – Experiments showed that 20 chunks strike a good balance between relevance and model overload; developers should validate this number for their own use case.
Reranking trade‑offs – Adding a reranker improves accuracy but adds latency and cost; parallel scoring mitigates latency, but the optimal top‑N to rerank depends on budget and response‑time requirements.

When a simple longer prompt suffices

If the knowledge base is **< 200 k tokens** (≈ 500 pages), Anthropic recommends embedding the entire corpus directly in the prompt, leveraging Claude’s prompt‑caching to keep latency low and cost down (> 2× speedup, up to 90 % cost reduction). Contextual Retrieval becomes valuable when the corpus exceeds this size.

How to get started

Anthropic provides a step‑by‑step cookbook that automates the entire workflow—from chunking and contextual metadata generation to embedding, BM25 indexing, and optional reranking. The cookbook is available at:

https://platform.claude.com/cookbook/capabilities-contextual-embeddings-guide

Takeaway

All of the evaluated techniques—embeddings + BM25, Contextual Retrieval, and reranking—stack additively. Deploying the full stack (Contextual Embeddings, Contextual BM25, reranking, and top‑20 chunk selection) yields the greatest reduction in retrieval failures and consequently the most reliable RAG‑driven applications.

Acknowledgements

Research and writing by Daniel Ford, with critical feedback from Orowa Sikder, Gautam Mittal, and Kenneth Lien; implementation support from Samuel Flamini; project coordination by Lauren Polansky; and editorial shaping by Alex Albert, Susan Payne, Stuart Ritchie, and Brad Abrams.