Marker-Inc-Korea/AutoRAG

What it solves

AutoRAG acts as a "librarian agent" for large document collections, moving beyond simple search tools that only provide file paths and matching lines. Instead of forcing the user to manually open files, read context, and synthesize answers, AutoRAG searches, reads, and curates information into clean, numbered knowledge units that provide direct answers to queries.

How it works

AutoRAG uses a two-tier agent architecture based on the Pi agent loop. A parent orchestrator handles planning, memory checks, and retrieval seeding, then delegates exploration tasks to specialized explorer agents. These explorers use tools like read, grep, find, and ls to inspect documents.

To ensure high-quality retrieval, it employs multiple pluggable methods:

• Lexical search: BM25 for keyword ranking.
• Semantic search: MinSync for understanding meaning.
• Hybrid search: A combination of both for precision and recall.
• Discovery: Optional integration with Jikji for find-first discovery and indexing.

It also features a self-evolving memory system that learns from user feedback and previous searches to optimize which retrieval methods and document areas are most productive for specific types of queries.

Who it’s for

It is primarily designed for people managing non-code document collections, such as manuals, legal documents, internal wikis, research papers, meeting notes, and PDFs.

Highlights

• Self-Evolving Memory: Learns from usage and explicit user feedback to improve search accuracy over time.
• Broad Datasource Support: Connects to external sources including Slack, Discord, Notion, GitHub, Google Drive, Gmail, and Obsidian.
• Multi-Method Retrieval: Combines BM25 and semantic search (MinSync) to the same interface.
• Structured Output: Returns curated knowledge units rather than raw search results.
• Legacy Support: Includes a WebAssembly parser for extracting text from legacy HWP5 binary files.