Is Grep All You Need? Analyzing Retrieval Strategies in Agentic Search

Grep Outperforms Vector Retrieval for Literal Information Recovery

In a comparative study of agentic search systems, literal string matching via grep generally yields higher accuracy than vector-based retrieval when the goal is to recover specific, stable witnesses such as exact dates, counts, and preferences. This finding suggests that for tasks requiring high precision and literal matching, traditional keyword search remains more effective than semantic embeddings.

The Impact of Agent Harnesses and Tool-Calling

While retrieval strategy is critical, the overall performance of an agentic search system depends heavily on the "harness"—the framework that manages how the model interacts with tools and how results are presented. The study compared several harnesses, including a custom harness called Chronos and provider-native CLI harnesses such as Claude Code, Codex, and Gemini CLI.

Key findings regarding harnesses include:

  • Harness Influence: Accuracy scores vary significantly based on the harness and tool-calling style used, even when the underlying conversation data remains identical.
  • Programming-Tuned Harnesses: Evidence suggests that harnesses optimized for programming tasks (e.g., Claude Code and Codex) show a stronger preference for and benefit from grep.
  • Neutral Harnesses: In contrast, neutral harnesses may see better results with vector search.

Experimental Methodology

The research was conducted through two primary experiments using the LongMemEval benchmark, which tests an agent's ability to answer questions over long conversations spanning multiple sessions.

  1. Experiment 1: Compared grep and vector retrieval across different harnesses (Chronos, Claude Code, Codex, Gemini CLI) and different tool-output formats (inline results versus file-based results read separately by the model).
  2. Experiment 2: Evaluated grep-only versus vector-only retrieval while progressively adding unrelated conversation history to test the model's ability to handle distracting material.

Critical Analysis and Community Perspectives

Technical discussions surrounding the study highlight several limitations and practical considerations regarding the reliance on grep for agentic search.

Benchmark Bias and Literalism

Critics argue that the results may be skewed by the choice of the LongMemEval benchmark. Because the benchmark rewards the recovery of "literal witnesses" (exact spans of text), it inherently favors grep over semantic search.

"A conversation about bikes, then a query about bike(s) where 'bike' is a common token hit. But not stuff like a conversation about a Beethoven sonata, then a question about classical music, where embedding based approach would shine."

Scalability and Resource Costs

While grep provides high accuracy for small to medium datasets, it introduces significant overhead in terms of token consumption and latency.

  • Token Consumption: grep often pulls larger chunks of surrounding text into the context window, increasing the cost per query.
  • Scaling Limits: Some researchers suggest that grep effectiveness breaks down once the corpus exceeds approximately 100,000 files, at which point more robust search engines (like BM25 or vector databases) become necessary.

The Case for Hybrid Approaches

Many practitioners argue that the choice between grep and vector search is a false dichotomy. Effective agentic workflows often benefit from a combination of tools:

  • Hybrid Search: Combining regex filtering with semantic ranking (e.g., using multi-vector embeddings) to capture both literal and conceptual matches.
  • Tool Autonomy: Allowing the agent to choose between grep, hybrid search, and structured data queries (such as graph frameworks or SQL) based on the specific nature of the user's request.
  • Semantic Mapping: Utilizing tools like Tree-Sitter, PageRank, or Language Server Protocols (LSP) to build semantic maps of codebases, which provides context that grep cannot capture.

Sources