stanford-oval/suql

What it solves

SUQL addresses the challenge of querying hybrid data sources that contain both structured (relational) and unstructured (free-text) information. Traditional SQL is effective for structured data, but cannot handle free-text search. SUQL extends SQL by adding free-text primitives to allow for precise and succinct representations of queries over hybrid databases.

How it works

SUQL integrates retrieval models, Large Language Models (LLMs), and traditional SQL. It uses a retrieval model and LLM for unstructured data and relational SQL for structured data. A key component is the answer function, which allows users to combine free-text constraints with structured constraints. To handle large-scale data, it uses PostgreSQL for scalability and FAISS for indexing free-text fields with dense vector processing methods like HNSW and product quantizer.

Who it’s for

Developers building conversational agents or chatbots that need to access hybrid data sources, such as a restaurant database containing both structured ratings and structured reviews.

Highlights

• Hybrid Data Access: Seamlessly combines SQL and free-text retrieval in a single query language.
• Scalability: Supports large databases via PostgreSQL and FAISS indexing.
• LLM Optimization: Includes optimizations to minimize expensive LLM calls.
• SQL Compatibility: Supports general SQL operations like JOINs and GROUP BYs.