MinishLab/semhash

What is SemHash?

SemHash is a lightweight Python library that helps you clean up large collections of data – text, images, audio, or any other modality – by:

Removing duplicate items (semantic deduplication)
Filtering out outliers that don’t fit the rest of the data
Choosing a small set of representative samples that capture the diversity of the whole collection.

It works by turning each record into a vector embedding (using the companion model2vec library for text, or any custom encoder you provide) and then performing fast similarity search with Vicinity (an ANN library). The API lets you operate on a single dataset or compare two datasets (e.g., to prevent train‑test leakage).

Why you might want it

Speed – Uses approximate‑nearest‑neighbor search, so even millions of records can be processed in seconds on a CPU.
Multimodal – Works out‑of‑the‑box for text and images; any modality can be added by supplying an encoder that follows a tiny encode protocol.
Explainable – After deduplication you can inspect which items were considered duplicates, see similarity scores, and retune thresholds without re‑running the whole pipeline.
Flexible data formats – Accepts plain Python lists, dictionaries, Pandas DataFrames, or Hugging‑Face datasets objects, and can handle multi‑column records (e.g., QA pairs).

Core concepts

Concept	What it does
Embedding	Turns raw data (text, image, etc.) into a fixed‑size numeric vector. SemHash uses `model2vec` for text by default, but you can plug in any encoder (Sentence‑Transformers, CLIP, custom vision models, etc.).
ANN backend	The similarity‑search engine. Default is USearch (via Vicinity) for high recall and speed; alternatives include FAISS or a simple exact‑match backend.
SemHash instance	Created with `SemHash.from_records(...)` (or from pre‑computed embeddings). Holds the dataset and the chosen encoder/backend.
Operations	`self_deduplicate`, `deduplicate`, `self_filter_outliers`, `filter_outliers`, `self_find_representative`, `find_representative`. Each returns a result object (`DeduplicationResult` or `FilterResult`).
Result objects	Provide `.selected` (kept records), `.filtered` (removed records), statistics (`duplicate_ratio`, etc.), and helper methods like `get_least_similar_from_duplicates` or `rethreshold`.

Quick‑start example (text)

from datasets import load_dataset
from semhash import SemHash

texts = load_dataset("ag_news", split="train")["text"]
semhash = SemHash.from_records(records=texts)

# 1️⃣ Remove duplicates
clean = semhash.self_deduplicate().selected

# 2️⃣ Drop outliers
clean = semhash.self_filter_outliers().selected

# 3️⃣ Pick a small, diverse subset
representative = semhash.self_find_representative().selected

All three steps are one‑liners once the SemHash object is built.

How it works under the hood (high‑level)

Embedding – Each record is passed through the supplied encoder → a dense vector.
Indexing – Vectors are inserted into Vicinity’s ANN index (USearch by default).
Similarity search – For each vector, the nearest neighbours are retrieved. A similarity threshold decides whether two items are duplicates or outliers.
Clustering – Duplicate items are grouped into clusters; the most central item is kept.
Representative selection – A greedy far‑thest‑point algorithm picks a subset that maximises coverage of the embedding space.

When to use SemHash

Data cleaning before training – Remove near‑duplicate sentences or images that could bias a model.
Leakage detection – Ensure your test set doesn’t contain items too similar to the training set.
Dataset summarisation – Create a small, diverse subset for quick prototyping or human annotation.
Cross‑modal pipelines – Any domain where you can embed items (audio, video, code) can be deduplicated with a custom encoder.

Limitations & things to keep in mind

Quality depends on the encoder – If the embedding model doesn’t capture semantics well, duplicates may be missed or false positives may appear.
Threshold tuning – The default similarity threshold works for many cases, but you may need to adjust it; the library provides tools (rethreshold, get_least_similar_from_duplicates) to help.
Memory – While ANN reduces compute, the full set of embeddings still lives in RAM; very large datasets may require batching or streaming.

Where to find more

Package – Install with pip install semhash (PyPI badge shows current version).
Source code – https://github.com/MinishLab/semhash
Docs & examples – The README contains extensive code snippets for text, images, custom encoders, Pandas, and using different ANN backends.
Community – Discord link in the badge for support and discussion.
Benchmarks – See benchmarks/README.md for performance numbers (e.g., 1.8 M text records in ~74 s on CPU).

TL;DR

SemHash is a fast, multimodal deduplication and filtering toolkit that turns your raw data into embeddings, finds near‑duplicates with an ANN index, and lets you inspect or retune the results. It’s useful for cleaning large training corpora, preventing data leakage, and extracting a representative subset, all with a small, dependency‑light Python package.