supervc-stack/VectorChord-bm25

What it solves

VectorChord-BM25 provides a native BM25 ranking algorithm for PostgreSQL, filling a gap where standard PostgreSQL full-text search (using tsvector and GIN indexes) lacks modern relevance scoring. It allows users to perform efficient, high-quality full-text search and ranking directly within their database without needing to offload search operations to external engines like Elasticsearch.

How it works

The project implements the Block-WeakAnd algorithm for BM25 ranking as a PostgreSQL extension. It introduces a specialized bm25vector data type—a sparse vector storing token IDs and their frequencies—and a corresponding bm25 index to accelerate the search and ranking process. It is designed to work alongside pg_tokenizer.rs to handle the conversion of raw text into these sparse vectors using either pre-trained models (like BERT) or custom-trained tokenizers for specific domains or languages (including support for Chinese via Jieba and Japanese via Lindera).

Who it’s for

Developers and database administrators using PostgreSQL who need professional-grade full-text search and relevance ranking (BM25) while maintaining a native database experience and avoiding the complexity of managing a separate search cluster.

Highlights

• Native PostgreSQL Integration: Implemented as a custom operator and index, providing a more intuitive API than external replacements.
• Flexible Tokenization: Supports pre-trained models, custom-trained models, and specialized pre-tokenizers for non-space-separated languages.
• High Performance: Utilizes the Block-WeakAnd algorithm to speed up the retrieval of the most relevant documents.
• Sparse Vector Storage: Uses a bm25vector type to efficiently store vocabulary IDs and term frequencies.