Anthropic Contextual Retrieval

Anthropic has introduced Contextual Retrieval, a preprocessing technique for Retrieval-Augmented Generation (RAG) that reduces retrieval failure rates by 49% when combining contextual embeddings and BM25, and by 67% when further integrated with reranking. This method solves the "context conundrum" where traditional RAG chunks lose the necessary background information to be accurately retrieved or utilized by an AI model.

The Context Conundrum in Traditional RAG

Traditional RAG systems typically break documents into small chunks (often a few hundred tokens) to maintain efficiency. However, this process often destroys critical context. For example, a chunk stating "The company's revenue grew by 3%" is useless if the system cannot determine which company or time period the chunk refers to, leading to retrieval failures when users ask specific questions about a particular entity.

How Contextual Retrieval Works

Contextual Retrieval solves the context loss problem by prepending a short, succinct explanatory context (typically 50-100 tokens) to each chunk before it is embedded or indexed.

Implementation via Claude

To avoid manual annotation, Anthropic uses Claude 3 Haiku to automatically generate this context. The model is provided with the entire document and the specific chunk, then prompted to provide a short context that situates the chunk within the overall document for search retrieval purposes.

The Preprocessing Pipeline

Context Generation: Claude generates a succinct context for each chunk based on the whole document.
Contextual Embeddings: The contextualized chunk is converted into a vector embedding.
Contextual BM25: The contextualized chunk is indexed using BM25 (Best Matching 25) for lexical matching.
Hybrid Search: The system combines results from both semantic embeddings and BM25 using rank fusion to find the most relevant chunks.

Performance Benchmarks

Anthropic tested this method across various domains, including codebases, fiction, and scientific papers. Using 1 minus recall@20 as the metric (measuring the percentage of relevant documents that fail to be retrieved), they found the following improvements:

Contextual Embeddings: Reduced the top-20-chunk retrieval failure rate by 35% (from 5.7% to 3.7%).
Contextual Embeddings + Contextual BM25: Reduced the failure rate by 49% (from 5.7% to 2.9%).
Contextual Retrieval + Reranking: Combining the above with a reranking step (using the Cohere reranker) reduced the failure rate by 67% (from 5.7% to 1.9%).

Cost and Latency Optimization

Prompt Caching

Generating context for every chunk can be expensive. Anthropic leverages prompt caching to reduce these costs. By caching the reference document once and referencing it for every chunk, the one-time cost to generate contextualized chunks is approximately $1.02 per million document tokens (based on 800 token chunks and 8k token documents).

Reranking Trade-offs

While reranking significantly boosts accuracy, it adds a runtime step that increases latency and cost. Developers must balance the number of chunks reranked (e.g., retrieving 150 and reranking down to 20) against the required response speed.

Key Implementation Recommendations

Based on their extensive testing, Anthropic provides the following guidelines for maximizing RAG performance:

Use Hybrid Search: Combining embeddings and BM25 is superior to using embeddings alone.
Select High-Performing Embeddings: Gemini and Voyage embeddings were found to be particularly effective.
Optimize Chunk Volume: Passing the top 20 chunks to the model generally performed better than passing 5 or 10.
Stack Techniques: The highest performance is achieved by combining contextual embeddings, contextual BM25, and a reranking step.