TAG-Research/TAG-Bench

TAG-Bench: A benchmark for table-augmented generation (TAG)

What it solves

It addresses the limitation of standard Text2SQL and RAG methods when answering natural language questions over databases, specifically when queries require world knowledge or semantic reasoning that isn't explicitly contained within the database itself.

How it works

The project introduces Table-Augmented Generation (TAG), a paradigm that allows for more complex interactions between a language model and a database. It provides a benchmark (TAG v1) consisting of 80 queries derived from the BIRD benchmark, split into 40 queries requiring parametric knowledge and 40 requiring reasoning. The benchmark evaluates various methods, including hand-written TAG, Text2SQL, RAG, and Retrieval + LM Rank, to measure accuracy and latency.

Who it’s for

Researchers and developers working on the intersection of AI and databases, specifically those looking to improve how LLMs interact with structured data for complex query answering.

Highlights

  • First benchmark specifically designed to study the Table-Augmented Generation problem.
  • Focuses on queries that require world knowledge and semantic reasoning beyond simple database retrieval.
  • Includes a comprehensive evaluation suite to compare TAG against Text2SQL and RAG.
  • Built upon the BIRD Text2SQL benchmark.

Related

  • Project
    GraphRAG-Bench/GraphRAG-Benchmark

    A comprehensive benchmark for evaluating Graph Retrieval-Augmented Generation (GraphRAG) models to determine when they outperform traditional RAG across various reasoning and retrieval tasks.

  • Project
    hhy-huang/HiRAG

    HiRAG is a Retrieval-Augmented Generation framework that uses hierarchical knowledge to improve the quality and comprehensiveness of LLM responses compared to standard RAG.

  • Project
    GiovanniPasq/agentic-rag-for-dummies

    A modular Agentic RAG framework using LangGraph that implements hierarchical indexing, multi-agent parallel retrieval, and human-in-the-loop query clarification.

  • Dispatch
    Evaluating RAG with LLM as a Judge

    Mistral AI outlines a framework for evaluating Retrieval-Augmented Generation (RAG) systems using an LLM-as-a-judge approach based on the RAG Triad metrics of context relevance, groundedness, and answer relevance.

  • Project
    rag-web-ui/rag-web-ui

    An intelligent dialogue system that allows users to build custom Q&A services by combining their own document knowledge bases with LLMs using Retrieval-Augmented Generation.