Marker-Inc-Korea/AutoRAG

What it solves

AutoRAG is a "librarian agent" designed to replace traditional search tools that only provide lists of file paths and matching lines. Instead of forcing the user to manually open files, read context, and synthesize answers, AutoRAG searches across various document collections, reads the source material directly, and curates a structured, numbered list of knowledge units (answers) that the user can act on immediately.

How it works

AutoRAG operates as a specialized search agent (based on the Pi agent loop) that manages the entire retrieval, reading, judgment, and curation loop using a single configured model. It employs several key mechanisms:

• Pluggable Retrieval: It supports multiple search strategies including literal pattern matching (grep), semantic vector search, and keyword ranking (BM25), often combining them into hybrid retrieval.
• Direct File Access: The agent uses a built-in bash tool to open and read source material directly from configured directories.
• Self-Evolving Memory: It learns from every search, tracking which retrieval methods and document areas are most productive for specific queries and incorporating user feedback to improve over time.
• Advanced PDF Handling: It uses a default parser with a fallback to a hybrid backend (docling-fast) for sparse or multi-page PDFs to ensure high-quality extraction.
• Datasource Skills: It can integrate with external data sources (e.g., Slack, Discord, Notion, GitHub, Gmail, Obsidian) via external CLI tools (like wacrawl, slacrawl, discrawl) to bring external communication and knowledge bases into the same retrieval pipeline.

Who it’s for

Users who manage large, fragmented document collections (PDFs, wikis, notes, research papers) and external communication channels (chat apps, email, cloud drives) and need a system that provides synthesized answers rather than just a list of search results.

Highlights

• Curated Answers: Returns structured knowledge units instead of raw file paths.
• Self-Improving: Learns retrieval strategies and productive document areas through usage.
• Multi-Modal Retrieval: Combines BM25, vector, and hybrid search for different document types.
• Extensive Datasource Integration: Connects to a wide array of platforms including KakaoTalk, WhatsApp, Telegram, Slack, Discord, Notion, GitHub, Gmail, and cloud drives via rclone.
• Duplicate Management: Integrates with dupey to detect and exclude exact duplicate documents during indexing.
• Optional Discovery: Can opt into Jikji for a find-first discovery and indexing layer.