spiceai/spiceai

🌶️ Spice.ai – Open‑source data‑grounded AI engine

What it is Spice.ai (the spiceai/spiceai repo) is a Rust‑based runtime that sits next to your application and lets you query, search, and run LLM inference over any data source with a single, familiar SQL‑style interface. It can materialise ("accelerate") selected tables locally for sub‑millisecond latency, or push queries down to the original source. All of this is exposed via standard APIs – HTTP, Arrow Flight, JDBC/ODBC, and an OpenAI‑compatible LLM gateway – so you can treat the engine like a tiny database that also knows how to call large language models.

Why it matters

• Real‑time analytics replica – CDC‑based, zero‑ETL replication of PostgreSQL, MySQL, MongoDB (and DynamoDB Streams) gives you an analytics‑ready copy of operational data with ~2 s freshness and no load on the production DB.
• Hybrid search at petabyte scale – Vector, full‑text, and combined (RRF) search are built‑in and can run directly on S3 objects, DuckDB HNSW, or Elasticsearch kNN indexes, all callable from SQL.
• AI‑native runtime – OpenAI‑compatible endpoints, Model Context Protocol (MCP) server/gateway, and a text‑to‑SQL (NSQL) feature let agents generate grounded queries automatically.
• Distributed query without heavy ops – Extends Apache Ballista with multi‑active schedulers coordinated via object storage, giving cluster‑scale performance with only a binary or container to run.
• Pluggable acceleration – The “Cayenne” engine (built on the Vortex columnar format) can keep accelerated tables in RAM or on disk, with secondary indexes, clustering, and point‑lookup optimisations.

Core components

Component	What it does	Access method
SQL & Search API	Standard SQL, vector/text search, UDTFs (vector_search, text_search, rrf, rerank)	HTTP, Arrow Flight, Flight SQL, JDBC/ODBC/ADBC
NSQL (text‑to‑SQL)	Turns natural‑language prompts into SQL against the federated schema	HTTP API, REPL, agent tool calls
LLM gateway (OpenAI‑compatible)	Routes calls to OpenAI, Anthropic, Bedrock, Vertex AI, Databricks, or locally‑served CUDA/Metal models; includes prompt caching	HTTP (OpenAI‑style)
Iceberg Catalog REST	Unified Iceberg catalog for query and ACID‑safe writes (INSERT INTO)	REST
MCP server/gateway	Implements the Model Context Protocol (2026‑07‑28 spec) for streaming model‑context interactions	HTTP + SSE

Typical workflow

1. Declare datasets – Write a spicepod.yaml that lists any of the 30+ supported sources (Postgres, Snowflake, S3, Kafka, GitHub, etc.).
2. Choose acceleration – Mark tables to be materialised in memory or on disk; configure CDC, indexing, clustering, etc.
3. Run the binary – spiced (the runtime) can be launched as a sidecar, a standalone service, or a multi‑node cluster.
4. Query from your app – Use any SQL client or call the HTTP/Flight endpoints; embed LLM calls directly in SQL or let an agent generate queries via NSQL.

Who would use it

• Product teams building AI‑augmented features that need up‑to‑date business data without building custom pipelines.
• Data engineers who want a low‑ops, federated query layer that can also serve vector search and LLM inference.
• ML ops / AI agents that require a single source of truth for both structured data and model calls, enabling “data‑grounded” agents.
• Enterprises needing multi‑tenant isolation, secret management (Vault, Azure Key Vault), mTLS, and observability via OpenTelemetry.

Getting started

# Install the CLI (binary or via Homebrew)
brew install spiceai/tap/spice
# Initialise a simple pod that connects to a local Postgres
spice init --from postgres://user:pass@localhost/db
# Start the runtime (runs as a sidecar)
spiced start
# Query via HTTP
curl http://localhost:8000/v1/sql -d 'SELECT * FROM orders LIMIT 5;'

The repo also provides a Cookbook (https://github.com/spiceai/cookbook) with step‑by‑step examples, and a Skills repository that lets AI coding agents auto‑configure Spice pods.

Performance highlights (as reported in the README)

• 2.9× faster than single‑node DataFusion on TPC‑H SF100.
• 8× less RAM than Spark with comparable query speed.
• Cayenne acceleration: 1.5× faster than DuckDB on TPC‑H SF100, 26× faster than Spice 1.x on TPC‑DS SF100.
• Hybrid search on S3 vectors: up to 90 % cost reduction vs. dedicated vector DBs.
• Real‑time CDC replication up to ~170× the ingest rate of Spice 1.x.

License Apache‑2.0 (open source). The repository contains the runtime (spiced) and CLI (spice). Commercial Cloud and Enterprise editions are optional add‑ons built on the same core.

---

Bottom line – Spice.ai is a full‑stack, open‑source data‑and‑AI engine that lets you treat any collection of databases, data lakes, and LLMs as a single, low‑latency queryable service. It removes the need for separate ETL pipelines, vector stores, and model‑hosting infrastructure, making it a compelling foundation for data‑grounded applications and AI agents.