visual-layer/fastdup
fastdup is a powerful, free tool designed to rapidly generate valuable insights from image and video datasets. It helps enhance the quality of both images and labels, while significantly reducing data operation costs, all with unmatched scalability.
What it solves
Fastdup is designed to manage, clean, and curate large-scale image and video datasets. It addresses the problem of dataset noise—such as duplicate images, near-duplicates, outliers, mislabeled data, and low-quality images (blurry, too dark, or too bright)—which can degrade the performance of AI models.
How it works
The tool uses an optimized C++ engine to analyze visual data locally or in the cloud. It can process labeled or unlabeled datasets and supports various embedding methods, including its own internal analysis, TIMM (PyTorch Image Models) and ONNX models (like DINOv2) to extract feature vectors for similarity search and dataset auditing.
Who it’s for
It is intended for data scientists and ML engineers who need to curate high-quality visual datasets at scale, supporting everything from small sets to billions of images on CPU-based infrastructure.
Highlights
- Extreme Scalability: Capable of processing 400 million images on a single CPU machine and scaling up to billions.
- Comprehensive Cleaning: Identifies duplicates, near-duplicates, outliers, and mislabels.
- Quality Auditing: Detects broken images and images with poor lighting or blur.
- Privacy-First: Runs locally or on private cloud infrastructure so data remains secure.
- Integration: Works with Hugging Face Datasets and supports custom ONNX embeddings.