xlang-ai/Spider2

[ICLR 2025 Oral] Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

What it solves

Spider 2.0 is a benchmark designed to evaluate the ability of Large Language Models (LLMs) to handle real-world, enterprise-level text-to-SQL workflows. Unlike previous benchmarks, it focuses on complex data environments—such as those with over 3,000 columns—and requires models to work across multiple SQL dialects including BigQuery and Snowflake.

How it works

The project provides three distinct evaluation settings to test different capabilities:

  • Spider 2.0-Snow: A text-to-SQL task using Snowflake databases.
  • Spider 2.0-Lite: A text-to-SQL task spanning BigQuery, Snowflake, and SQLite.
  • Spider 2.0-DBT: A code agent task using DuckDB (DBT) that focuses on repository-level text-to-SQL tasks.

To facilitate benchmarking, the project includes the "Spider-Agent" framework, which can be implemented via Docker or a faster tool-call-based version for rapid testing.

Who it’s for

It is primarily for AI researchers and developers building or evaluating LLMs for code generation, specifically those focusing on complex database interactions and enterprise SQL workflows.

Highlights

  • Enterprise Scale: Tests models on massive schemas with thousands of columns.
  • Multi-Dialect Support: Covers BigQuery, Snowflake, and SQLite.
  • Diverse Task Types: Includes both standard text-to-SQL and agentic code generation tasks (DBT).
  • Real-World Complexity: Specifically designed to be significantly more challenging than previous benchmarks like Spider 1.0 and BIRD.

Related

  • Dispatch
    Enterprise Scenarios Leaderboard: Evaluating LLMs for Real-World Use Cases

    Hugging Face and Patronus AI have launched the Enterprise Scenarios Leaderboard to evaluate language models on six real-world business tasks, moving beyond academic benchmarks to measure practical enterprise utility.

  • Dispatch
    Senior SWE-Bench Open-Source Benchmark for Evaluating LLMs as Senior Engineers

    Senior SWE-Bench is an open-source benchmark that measures how well AI agents perform senior software engineering tasks, highlighting both current capabilities and challenges of LLM-driven development.

  • Project
    vllm-project/guidellm

    An SLO-aware benchmarking and evaluation platform that simulates real-world workloads to optimize LLM inference performance, resource needs, and operational limits.

  • Project
    SWE-bench/SWE-bench

    A benchmark for evaluating large language models on their ability to resolve real-world GitHub software issues by generating functional code patches.

  • Project
    NVIDIA/RULER

    A benchmarking framework that evaluates the real effective context size of long-context LLMs using synthetic tasks that go beyond simple retrieval.