ldqk/ImageSearch

基于.NET10的本地硬盘千万级图库以图搜图案例Demo和相似图片查重和图片exif信息移除小工具分享，全面对标DuplicateCleaner

What it solves

This project provides a high-performance local tool for searching and managing large collections of images and videos. It solves the problem of finding duplicate or visually similar files across massive local datasets, offering faster performance than traditional tools like DuplicateCleaner.

How it works

It uses perceptual hashing algorithms (Difference Hash, DCT Hash 32, and DCT Hash 64) to create a searchable index of your local files. For videos, it extracts keyframes at one-second intervals, scales them down, and hashes them for comparison. Users can adjust similarity thresholds and select specific algorithms to fine-tune search results. The tool can optionally integrate with the 'Everything' search engine to accelerate file discovery.

Who it’s for

It is designed for Windows users who need to organize, deduplicate, or search through large local media libraries without relying on cloud-based services.

Highlights

  • Supports rapid indexing and searching of millions of images and videos.
  • Includes advanced deduplication with automated selection rules (e.g., keep newest, keep largest).
  • Provides flexible search options, including clipboard search and drag-and-drop support.
  • Features built-in EXIF data removal capabilities.
  • Offers visual previews and batch management tools for cleaning up disk space.

Related

  • Project
    NGT-labs/NGT

    NGT is an open‑source library for fast approximate nearest‑neighbor search in high‑dimensional vector spaces. It provides graph‑ and tree‑based indexes plus quantized variants (QG, QBG) that scale to billions of items, supports many distance metrics, offers language bindings (Python, Ruby, PHP, Rust, Go, Node.js), and can run as a standalone CLI or via distributed servers (ngtd, vald). Licensed under Apache 2.0.

  • Project
    datascale-ai/data_engineering_book

    An open-source, comprehensive guide to LLM data engineering that provides a systematic framework and 15 practical projects for managing data pipelines from pre-training to production.

  • Project
    xtreme1-io/xtreme1

    Xtreme1 is an open‑source, Docker‑based platform for annotating, curating, and managing multimodal training data (2‑D images, 3‑D LiDAR, sensor‑fusion). It offers UI‑driven labeling tools, ontology management, built‑in pre‑labeling models (YOLOR, OpenPCDet), quality monitoring, and a beta RLHF interface for LLMs. Installable via a single `docker compose up` command, it runs on any OS with Docker and can optionally use GPU‑accelerated model containers. Licensed under Apache‑2.0 and hosted by the LF AI & Data Foundation.

  • Project
    amphi-ai/amphi-etl

    Amphi ETL is a visual, low-code tool that helps users build data preparation and transformation pipelines by generating native Python code. It is designed to be self-hosted, ensuring data privacy while simplifying the creation of pipelines for reporting and lightweight ETL tasks.