Bergvca/string_grouper

Super Fast String Matching in Python

What it solves

It simplifies and accelerates the process of finding and grouping similar strings within one or more lists. This is particularly useful for deduplicating large datasets where strings may have slight variations (fuzzy matching) but represent the same entity.

How it works

The library uses tf-idf to calculate cosine similarities between strings. To achieve high performance on large datasets, it leverages a Rust-based sparse matrix multiplication backend (sp_matmul_rs), allowing it to process hundreds of thousands of strings in seconds.

Who it’s for

Data scientists and developers working with large-scale text datasets who need to perform efficient fuzzy matching and string deduplication.

Highlights

  • High Speed: Capable of fuzzy matching 663,000 names in under 18 seconds on consumer hardware.
  • Graph-based Grouping: Able to resolve indirect associations between strings to form cohesive groups.
  • Centroid Identification: Determines a representative "centroid" string for each group.
  • Flexible Backends: Supports both a modern Rust implementation and the original sparse_dot_topn backend.

Related

  • Project
    ing-bank/sparse_dot_topn

    A fast, parallelized sparse matrix multiplication library that integrates top-n result selection to reduce memory usage and improve performance when matching large feature vectors.

  • Project
    klaudiosinani/moviebox

    A content-based movie recommendation system that uses TF-IDF and cosine similarity to suggest movies with similar plot summaries.

  • Project
    dmtrKovalenko/fff

    A high-performance file search toolkit for humans and AI agents that provides typo-resistant, frecency-ranked path and content search via an in-memory index.

  • Project
    facebookresearch/faiss

    A library for efficient similarity search and clustering of dense vectors, capable of handling billion-scale datasets using CPU and GPU acceleration.

  • Project
    castorini/pyserini

    Pyserini is a Python toolkit that wraps the Anserini Lucene search engine and FAISS to provide easy, reproducible first‑stage retrieval (sparse, dense, or hybrid) for information‑retrieval research and RAG pipelines. It includes pre‑built indexes, evaluation scripts, and a REST server, and can be installed via `pip`.