Anthropic Contextual Retrieval: Improving RAG Accuracy with Contextual Embeddings and BM25

Contextual Retrieval dramatically improves Retrieval-Augmented Generation (RAG) accuracy by preserving document context during the retrieval process.

Traditional RAG solutions often fail to retrieve relevant information because the process of chunking documents and creating embeddings removes crucial context. Contextual Retrieval solves this by prepending chunk-specific explanatory context to each chunk before it is embedded or indexed.

The limitations of traditional RAG systems

Retrieval-Augmented Generation (RAG) is a standard method for enhancing AI model knowledge by retrieving relevant information from a knowledge base and appending it to the user's prompt. However, traditional RAG systems suffer from two primary limitations: the loss of context and the inability to capture precise matches.

When documents are broken down into smaller chunks for efficient retrieval, these individual chunks often lack sufficient context. For example, a chunk stating, "The company's revenue grew by 3% over the previous quarter," is ambiguous without knowing which company or time period it refers to.

Furthermore, while semantic embeddings excel at capturing broad semantic relationships, they can miss crucial exact matches. Techniques like BM25 (Best Matching 25), which uses lexical matching, are necessary to find precise word or phrase matches, such as an exact error code like "TS-999."

Traditional RAG systems can combine these methods by creating TF-IDF encodings and semantic embeddings, using BM25 for exact matches, and then combining the results using rank fusion. However, this approach still suffers from the context conundrum.

Contextual Retrieval: A solution to the context problem

Contextual Retrieval is a preprocessing technique that enhances retrieval accuracy by addressing the context loss inherent in standard RAG. It achieves this through two sub-techniques: Contextual Embeddings and Contextual BM25.

Contextual Embeddings work by instructing a model (such as Claude) to generate concise, chunk-specific explanatory context for each chunk based on the overall document. This resulting contextual text, typically 50-100 tokens, is prepended to the original chunk before it is embedded or indexed.

This transformation allows the retrieval system to understand the context surrounding the chunk, ensuring that even small chunks are meaningful when retrieved.

Performance improvements of Contextual Retrieval

Experimental results demonstrate that Contextual Retrieval provides significant improvements in retrieval accuracy across various knowledge domains, including codebases, fiction, and scientific papers.

Contextual Embeddings alone reduced the top-20-chunk retrieval failure rate by 35% (from 5.7% to 3.7%).
Combining Contextual Embeddings and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 49% (from 5.7% to 2.9%).

Contextual Retrieval is uniquely deployable at a low cost when using Claude's prompt caching feature. Assuming 800 token chunks, 8k token documents, 50 token context instructions, and 100 tokens of context per chunk, the one-time cost to generate contextualized chunks is $1.02 per million document tokens.

Further boosting performance with Reranking

Retrieval accuracy can be further optimized by combining Contextual Retrieval with a reranking step. Reranking is a filtering technique that ensures only the most relevant chunks are passed to the generative model, which reduces cost and latency.

The process involves:

Initial retrieval of a large set of potentially relevant chunks (e.g., top 150).
Passing these chunks and the user query through a reranking model.
Scoring each chunk based on its relevance and importance.
Selecting the top-K chunks (e.g., top 20) to pass as context.

When Reranked Contextual Embedding and Contextual BM25 are used, the top-20-chunk retrieval failure rate is reduced by 67% (from 5.7% to 1.9%).

Implementation considerations

Developers implementing Contextual Retrieval should consider several factors to maximize performance:

Chunk Boundaries: The choice of chunk size, chunk boundary, and chunk overlap significantly affects retrieval performance.
Embedding Model: While Contextual Retrieval improves performance across all tested embedding models, Gemini and Voyage embeddings were found to be particularly effective.
Custom Contextualizer Prompts: Although a generic prompt is provided, tailoring the prompt to a specific domain or use case (e.g., including a glossary of key terms) may yield better results.
Number of Chunks: While more chunks increase the chance of including relevant information, adding too much information can distract the model. Testing showed that using 20 chunks was the most performant option among 5, 10, and 20 chunks.

In summary, the most effective approach to maximize performance is to combine contextual embeddings (from Voyage or Gemini) with contextual BM25, followed by a reranking step, and passing the top 20 chunks to the prompt.