VexDB-THU/VexDB-Lite

A cross-platform vector database, which can be integrated into existing databases as a plugin.

What it solves

VexDB-Lite provides high-performance vector similarity search capabilities for popular database engines including PostgreSQL, DuckDB, and SQLite. It addresses the need for efficient Approximate Nearest Neighbor (ANN) search and vector storage within existing database ecosystems, offering alternatives to standard vector extensions with better performance and compression options.

How it works

The project implements a custom graph-based index algorithm shared across all three database backends. To optimize performance and memory usage, it employs:

  • SIMD Distance Dispatch: Compile-time optimized, inlined distance functions for faster calculations.
  • Quantization: Supports Product Quantization (PQ) and RaBitQ to compress vectors, reducing memory footprints and allowing for "compact" storage modes where only quantized codes are kept.
  • Vector Buffer Cache: A shared-memory cache to accelerate vector access and reduce disk I/O.
  • Database Integration: It integrates as an extension (PostgreSQL/DuckDB) or a virtual table (SQLite), allowing users to perform vector searches using standard SQL queries with distance operators like L2, cosine, and inner product.

Who it’s for

  • Database administrators and developers who want to add vector search to PostgreSQL, DuckDB, or SQLite.
  • AI engineers building RAG (Retrieval-Augmented Generation) systems who require a high-performance vector store integrated into their relational database.
  • Developers targeting edge or mobile platforms (iOS, Android, WASM) who need efficient vector search via SQLite.

Highlights

  • Multi-Backend Support: Single core engine supporting PostgreSQL, DuckDB, and SQLite.
  • Advanced Quantization: Includes RaBitQ and PQ for significant memory reduction.
  • High Throughput: Benchmarks show higher QPS compared to pgvector and duckdb-vss.
  • Broad Platform Compatibility: Prebuilt packages for Linux, macOS, iOS, Android, and WASM.

Related

  • Project
    1yefuwang1/vectorlite

    A runtime-loadable SQLite extension that enables fast, SIMD-accelerated vector search using HNSW, providing a SQL interface for approximate nearest neighbor queries.

  • Project
    alibaba/AliSQL

    An open-source MySQL branch that integrates DuckDB for high-performance analytics and native HNSW vector search for AI applications.

  • Project
    pgvector/pgvector

    An open-source PostgreSQL extension that enables vector similarity search, allowing users to store embeddings and perform nearest neighbor queries within their relational database.

  • Project
    timescale/pgvectorscale

    A PostgreSQL extension that enhances pgvector with a StreamingDiskANN index and Statistical Binary Quantization for high-performance, cost-efficient embedding search.

  • Project
    sqliteai/sqlite-vector

    A cross-platform SQLite extension that adds production-grade vector search and quantization to SQLite, enabling efficient semantic search and RAG on the edge and mobile devices.