xhluca/bm25s
Fast BM25 search in Python, powered by Numpy and Numba
What it solves
bm25s is an ultrafast Python implementation of the BM25 ranking function, used for lexical search and text retrieval. It addresses the performance bottlenecks of existing Python libraries by providing a significantly faster way to rank documents based on a query, making it a viable alternative to heavy search services like Elasticsearch for many use cases.
How it works
The library leverages Numpy and sparse matrices to store eagerly computed scores for all document tokens. This approach allows for extremely fast scoring at query time. It also supports optional JIT compilation via numba for further speedups on larger datasets and provides a memory-efficient mmap (memory-mapping) option to load indices without consuming full system memory.
Who it’s for
It is designed for developers building search services, RAG (Retrieval-Augmented Generation) pipelines, or any application requiring fast, keyword-based document retrieval in Python without the overhead of Java or PyTorch.
Highlights
- High Performance: Claims orders of magnitude speedup over popular Python BM25 implementations.
- Low Dependency: Pure Python powered by Numpy; no Java or PyTorch required.
- Flexible API: Supports various BM25 variants (Robertson, ATIRE, BM25L, BM25+, Lucene) and customizable tokenization.
- Tooling: Includes a command-line interface (CLI) for indexing and searching, and a built-in Model Context Protocol (MCP) server to expose indices as tools for LLMs.
- Ecosystem Integration: Native support for saving and loading indices via the Hugging Face Hub.