beowolx/rensa

What it solves

Rensa provides a high-performance way to estimate similarity between large sets of data and identify near-duplicate documents. It is designed to be a significantly faster alternative to existing libraries like datasketch and FastSketch for large-scale deduplication tasks.

How it works

Rensa implements MinHash, a technique that uses random hash functions to create a compact "signature" of a set. The fraction of matching slots between two signatures estimates the Jaccard similarity of the original sets.

Rensa offers two variants:

• R-MinHash: A custom variant that uses multiply-shift hashing (taking the upper 32 bits of a 64-bit multiply-add) instead of modular reduction, reducing CPU overhead and memory usage.
• C-MinHash: Based on a formal paper, this variant uses a two-stage scheme to derive multiple slots from a smaller parameter set, providing tighter variance bounds.

The project is written in Rust with Python bindings and employs several low-level optimizations, including non-cryptographic hashing, batch processing of elements, and the use of the MiMalloc allocator to handle memory more efficiently.

Who it’s for

Data engineers and ML practitioners who need to perform near-duplicate detection or similarity estimation on massive datasets where standard Python implementations are too slow.

Highlights

• Extreme Speed: Benchmarked as up to 608x faster than datasketch and 11x faster than FastSketch.
• Memory Efficient: R-MinHash signatures use 32-bit integers, halving memory requirements compared to 64-bit implementations.
• LSH Integration: Includes Locality Sensitive Hashing (LSH) for efficient candidate querying without comparing every pair.
• Streaming Support: Built-in deduplicator for continuous data streams.
• Flexible API: Supports both one-shot batch processing and incremental updates.