spiceai/spiceai

Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.

What it solves

Spice is a portable, accelerated SQL query, search, and LLM-inference engine designed to eliminate the need for complex data pipelines and "glue" code when building data-grounded AI applications and agents. It solves the problem of high latency and operational overhead when querying federated data sources, providing a way to run millisecond-latency queries and AI inference on localhost while scaling to a distributed cluster.

How it works

Spice operates using a "Cluster-Sidecar Architecture." A lightweight Spice sidecar runs in the application pod to serve SQL, search, and LLM-inference from a scoped working set on localhost. For larger datasets or "long tail" queries, it transparently delegates requests to a central Spice cluster powered by Apache Ballista for distributed query processing and the Spice Cayenne data accelerator. It supports native Change Data Capture (CDC) for real-time analytics replicas of PostgreSQL, MySQL, and MongoDB without requiring ETL tools like Kafka or Debezium.

Who it’s for

It is intended for developers building AI agents and data-intensive applications that require real-time access to federated data across various sources (e.g., S3, Snowflake, MongoDB, Postgres) and need to integrate LLM inference and vector search into a single query plan.

Highlights

  • Real-time Analytics Node: Creates sandboxed replicas of operational databases (Postgres, MySQL, MongoDB) with ~2-second freshness and zero production load.
  • AI-Native Runtime: Integrates OpenAI-compatible APIs, text-to-SQL (NSQL), and vector search (including Amazon S3 Vectors) directly into SQL primitives.
  • Hybrid Search: Combines BM25 full-text search, vector search, and reranking using Reciprocal Rank Fusion (RRF) in a single SQL query.
  • High-Performance Acceleration: Uses the Spice Cayenne accelerator on Vortex for faster random access and lower memory usage compared to Parquet or DuckDB.
  • Federated Querying: Connects to 30+ data sources with advanced query push-down capabilities.
  • Enterprise Ready: Includes mTLS, HashiCorp/Azure Key Vault integration, and Model Context Protocol (MCP) support.

Related

  • Project
    lucasjinreal/Crane

    Crane is a Rust‑based inference framework (built on candle) that runs LLMs, vision‑language models, OCR, ASR, TTS and music transcription fast on CPU, CUDA, Metal and experimental ROCm. It loads GGUF checkpoints directly, provides a clean Rust API and an OpenAI‑compatible HTTP server, and ships ready‑to‑use demo binaries for chat, image chat, OCR, speech, etc.

  • Project
    rtk-ai/icm

    ICM (Infinite Context Memory) is a Rust‑based, zero‑dependency binary that provides a shared, searchable SQLite memory store for AI agents. It supports episodic memories with importance‑based decay, permanent knowledge‑graph memoirs, feedback recording, and verbatim session transcripts. Integrated via MCP hooks or a CLI, it works with many popular assistants (Claude, Gemini, Codex, Copilot, etc.) and offers optional ONNX‑Runtime semantic search. Install via Homebrew, a one‑liner script, or Cargo; configure globally or per‑project, then store/recall context across tools without re‑explaining. Experimental (pre‑1.0) but usable today.

  • Project
    kkokosa/dotLLM

    A high-performance LLM inference engine written natively in C#/.NET, providing native CPU/GPU support for transformer models without relying on Python or llama.cpp.

  • Project
    EricLBuehler/mistral.rs

    A fast, flexible LLM inference engine written in Rust that supports multimodal models, agentic tool execution, and OpenAI/Anthropic compatible APIs.

  • Project
    Avarok-Cybersecurity/atlas

    A pure Rust and CUDA LLM inference engine that provides a lightweight, high-throughput OpenAI-compatible server without requiring Python or PyTorch.