RAG Architectures: Avoiding Over-Engineering in Retrieval Augmented Generation

Retrieval Augmented Generation (RAG) is frequently over-engineered, with developers jumping directly to embeddings and vector databases when simpler information retrieval methods are more effective. The optimal architecture depends on data freshness, corpus characteristics, query patterns, and scale, but the majority of systems can be successfully implemented using full-text search combined with LLM-based query rewriting.

Decision Framework for RAG Architectures

Choosing the right retrieval strategy requires evaluating five key technical factors to avoid unnecessary complexity:

Data Freshness: Real-time updates favor easy re-indexing; stable corpora allow for pre-embedding.
Corpus Characteristics: High churn (over 10% daily change) makes full pre-embedding impractical.
Query Patterns: Keyword-heavy queries require full-text search, while conversational queries benefit from embeddings.
Scale & Performance: Systems with fewer than 1,000 queries per day generally do not require full optimization.
Team Capabilities: Teams without ML expertise should prioritize full-text search and query rewriting over hybrid or advanced embedding pipelines.

RAG Implementation Recipes

Retrieval strategies should be implemented incrementally, moving to more complex architectures only when data proves the simpler method is insufficient.

1. Full-Text Search (BM25)

Full-text search using tools like Elasticsearch or Postgres is the most efficient starting point for keyword-style queries and exact matches (e.g., "invoice #12345").

Advantages: Zero API costs, sub-10ms latency, easy debugging, and no requirement for chunking strategies or complex evaluation.
Disadvantages: Fails to capture synonyms or semantic intent (e.g., "car" vs "automobile").

2. Full-Text Search with Query Rewriting

Using an LLM to transform conversational user queries into clean keyword searches solves many "semantic search" problems by addressing query formulation rather than retrieval.

Mechanism: An LLM removes stopwords, adds synonyms, translates domain-specific jargon, and decomposes complex queries.
Benefit: If results are poor, developers can adjust the system prompt rather than re-embedding the entire corpus. This is particularly effective for proprietary terminology that general-purpose embedding models often misinterpret.

3. Hybrid Search (BM25 + Embedding Reranking)

This approach uses BM25 to retrieve a broad set of candidates (top 50-100) and then uses embeddings to rerank the top 10.

Trade-off: This adds 200-500ms of latency but captures semantic meaning that keyword search misses.
Complexity: Introduces the need for a chunking strategy (fixed-size vs. semantic) and overlap management.

4. On-the-Fly Embedding

For high-churn data (over 10% daily updates) or real-time content, documents are embedded during the query process rather than upfront.

Advantages: Perfect data freshness and trivial model switching; changing embedding models requires only a one-line code change rather than re-indexing millions of documents.
Disadvantages: Higher query latency (200-500ms) and limited to small reranking sets (K=20-50).

5. Hot/Cold Tiering

This architecture pre-embeds frequently accessed documents ("hot tier") and embeds rarely accessed documents on-the-fly ("cold tier").

Benefit: Optimizes the Pareto distribution of access patterns (where 20% of docs get 80% of traffic), balancing latency and flexibility.

6. Full Pre-Embedding

Embedding the entire corpus upfront and storing it in a vector database for Approximate Nearest Neighbor (ANN) search is only justified for massive scale (over 10k queries/day) and very stable corpora.

Risk: Model deprecation is a significant burden. Switching models requires re-embedding the entire corpus, which involves high compute costs, downtime, and extensive regression testing.

Agentic RAG and Query Decomposition

Complex user queries containing multiple intents (e.g., "Read a CSV, clean data, and plot results") should be decomposed into sub-queries. An agentic system breaks the query down, routes each sub-query to the optimal retrieval method (e.g., simple keyword search for some, embeddings for others), and combines the results.

This decomposition is often significantly cheaper and more accurate than attempting to rewrite a single complex query and embedding a large set of documents.

Community Insights and Counterpoints

Industry practitioners emphasize that the operational burden of vector search is often underestimated.

"Semantic similarity isn’t as good as you think... You will inevitably end up having to re-embed more or different chunks of your text to accommodate more and more precise embedding search... then you turn around and build a search query with 500 keywords and sure it’s painful but it just works."

Other experts suggest that for specific domains like coding, retrieval can be counterproductive if it returns misleading chunks that the model trusts over the actual source code. In such cases, simple tools like grep or ripgrep combined with an LLM agent can be more reliable than a vector-based RAG pipeline.

Summary of Implementation Paths

Strategy	Freshness	Complexity	Latency	Best For
Full-Text + Rewriting	Perfect	Low	<50ms	60% of use cases
On-the-Fly Embedding	Perfect	Low	200-500ms	High churn data
Hot/Cold Tiers	Mixed	Medium	50-100ms	Varied access patterns
Full Pre-Embedding	Stale	High	<50ms	Massive scale, stable data