onyx-dot-app/EnterpriseRAG-Bench

Dataset and benchmark for RAG on company internal documents.

What it solves

Most Retrieval Augmented Generation (RAG) benchmarks rely on public data like Wikipedia or Stack Overflow. This project provides a realistic, large-scale dataset specifically designed to simulate company internal knowledge, allowing developers to benchmark RAG systems and fine-tune agents on the types of messy, fragmented, and jargon-heavy data found in actual enterprise environments.

How it works

The project centers around a simulated company called "Redwood Inference." It provides a corpus of over 500,000 documents across nine common enterprise sources, including Slack, Gmail, Google Drive, Jira, and Confluence. To ensure realism, the dataset incorporates:

  • Cross-document coherence: Documents share a common foundation of company initiatives and employee directories.
  • Realistic noise: The data includes outdated information, misfiled documents, and conflicting facts.
  • Internal terminology: The use of project codenames and company-specific acronyms.
  • Diverse Question Set: 500 questions categorized by difficulty and type, ranging from simple factual lookups to complex reasoning across multiple documents or handling conflicting information.

Who it’s for

AI engineers and researchers building RAG systems or AI agents for the enterprise who need a high-fidelity environment to test retrieval accuracy and reasoning capabilities on internal corporate data.

Highlights

  • Massive Scale: Over 500,000 documents simulating a full corporate ecosystem.
  • Multi-Source Simulation: Covers a wide array of tools including CRM (Hubspot), project management (Linear), and meeting transcripts (Fireflies).
  • Complex Evaluation: Includes specific test cases for conflicting information, intra-document reasoning, and "info not found" scenarios.
  • Dataset Generator: Provides code to generate similar synthetic datasets for different industries or company scales.