yanyiwu/simhash

中文文档simhash值计算

What it solves

This project provides a tool to calculate Simhash values for Chinese text. Simhash is a widely used algorithm for text deduplication, allowing you to identify similar or duplicate documents by comparing their generated "fingerprints."

How it works

The library processes Chinese text by using CppJieba for word segmentation and keyword extraction. It then applies the Jenkins hash function to these keywords to generate a 64-bit Simhash value. By comparing the Hamming distance between two Simhash values, you can determine if the documents are similar based on a configurable threshold.

Who it’s for

It is designed for developers working with Chinese text processing who need a lightweight, header-only C++ solution for identifying duplicate or near-duplicate content.

Highlights

  • Header-only (.hpp) design for easy integration without complex linking.
  • Built-in keyword extraction using CppJieba.
  • Automatic topN keyword selection based on text length for optimal performance.
  • Includes benchmarking tools to measure processing speed.

Related

  • Project
    lhotse-speech/lhotse

    A Python library for flexible multimodal data preparation that optimizes the loading and manipulation of speech, audio, video, and image data for ML training.

  • Project
    mongodb-js/mongodb-mcp-server

    MongoDB MCP Server is a Model Context Protocol server that lets AI agents safely run MongoDB queries and Atlas management commands. It ships plugins for popular AI tools, can be started via `npx` or Docker, defaults to read‑only, and supports a wide range of database and Atlas operations.

  • Project
    MIC-DKFZ/batchgenerators

    A Python package for 2D and 3D data augmentation that provides specialized tools for medical image computing and multi-threaded data loading.

  • Project
    OvertureMaps/data

    Overture Maps provides open-source, cloud-native global map data partitioned by theme and stored in Parquet format for efficient, large-scale spatial analysis.