skyzh/vector-db-from-scratch

skyzh/vector-db-from-scratch – Build a Vector Database from the Ground Up (Rust)

What it is

• An open‑source, hands‑on tutorial that walks systems or database engineers through building a tiny, in‑memory vector‑search engine in Rust.
• The material is delivered as a multi‑day course (and accompanying book) that gradually adds database‑style features such as Arrow‑backed tables, DataFusion optimizer rules, and several ANN index structures.

Why it matters

• Vector search is a core component of modern AI systems (e.g., similarity search for embeddings, retrieval‑augmented generation). This repo shows how those algorithms become first‑class database operators rather than black‑box services.
• By implementing the indexes yourself you see the trade‑offs between exact and approximate search, and you get a concrete benchmark suite (recall, build time, latency on the SIFT1M dataset).

Core learning outcomes

Day	Feature added
1	Arrow‑backed in‑memory table + safe DataFusion optimizer rule
2	Exact (brute‑force) vector search
3	IVFFlat index
4	NSW (Navigable Small World) index
5	HNSW index
6	Residual IVF‑PQ index
–	SQL commands to create indexes, compare query plans, and run benchmarks

How you use it

1. Clone the repo and follow the guided Rust notebooks linked from the course site.
2. Each day provides starter code, unit tests, and a reference implementation you can compare against.
3. After building the indexes you can issue SQL statements (via DataFusion) that invoke the vector search directly, letting you blend vector similarity with traditional relational queries.
4. Run the shared benchmark to see how recall, build time, and query latency vary across the different index types.

Audience

• Engineers comfortable with Rust (no prior vector‑search or DataFusion knowledge required).
• Anyone interested in the systems side of AI—how embedding‑based retrieval is integrated into databases.

Licensing

• Code: Apache License 2.0.
• Book/markdown/figures: Creative Commons BY‑NC‑SA 4.0.
• Submodules (bustub-vectordb-starter / bustub-vectordb-solution) keep their original licenses.

Community

• A Discord server is provided for learners to discuss exercises, share results, and ask questions.

Bottom line skyzh/vector-db-from-scratch is a genuine, educational open‑source project that teaches you to build the fundamental pieces of a vector database—an increasingly important building block for AI‑driven applications—using Rust and the DataFusion query engine.