Image Similarity with Hugging Face Datasets and Transformers

Hugging Face has provided a technical guide on building an image similarity system using the Transformers and Datasets libraries. This system allows users to identify the most similar images from a set of candidates given a query image, a core functionality for information retrieval systems like reverse image search.

Defining Image Similarity via Embeddings

Image similarity is achieved by converting high-dimensional pixel data into dense representations called embeddings. This process compresses the image space (e.g., 224 x 224 x 3 pixels) into a lower-dimensional vector space (e.g., 768 dimensions), which significantly reduces computation time during retrieval.

To determine the similarity between two images, the system computes the cosine similarity metric between their respective embeddings. The closer the cosine similarity score, the more similar the images are considered to be in the vector space.

Computing Embeddings with Image Encoders

To generate these embeddings, a vision model acting as an image encoder is required. Hugging Face recommends using the AutoModel class to load compatible checkpoints from the Hugging Face Hub.

Model Selection

In the provided example, the system uses nateraw/vit-base-beans, a Vision Transformer (ViT) model fine-tuned on the beans dataset. The guide highlights several key technical choices:

  • AutoModel vs AutoModelForImageClassification: AutoModel is used because the goal is to obtain dense representations (embeddings) rather than discrete category labels.
  • Domain-Specific Fine-tuning: Using a model fine-tuned on the specific dataset (like beans) generally results in better understanding and retrieval performance than using a generalist model trained on ImageNet-1k.
  • Self-Supervised Learning: The guide notes that checkpoints obtained through self-supervised pre-training can also yield impressive retrieval performance.

Alternative Models

While the example uses ViT, the system can be extended to other vision models available in the Transformers library, including:

  • Swin Transformer
  • ConvNeXT
  • RegNet

The Image Similarity Workflow

The process of finding similar images follows a four-step pipeline:

  1. Candidate Embedding Extraction: Embeddings are extracted from a candidate image subset and stored in a matrix.
  2. Query Embedding Extraction: The query image is processed through the same image encoder to generate its embedding.
  3. Similarity Scoring: The system iterates over the candidate embedding matrix and computes the cosine similarity between the query embedding and each candidate.
  4. Ranking: The results are sorted by similarity score, and the top-k identifiers are returned to fetch the most similar images.

To implement this efficiently, the map() function from the datasets library is used to compute embeddings in parallel across the candidate dataset.

Scaling for Large Datasets

For large-scale production environments involving millions of images, storing raw 768-dimensional embeddings creates memory and computational bottlenecks. Hugging Face suggests two primary ways to optimize this:

Dimensionality Reduction

Reducing the dimensionality of embeddings without losing their semantic meaning can balance speed and retrieval quality. Techniques such as random projection and locality-sensitive hashing (LSH) are recommended for this purpose.

FAISS Integration

For high-performance similarity search, the datasets library provides direct integration with FAISS (Facebook AI Similarity Search). This integration simplifies the indexing process:

  • add_faiss_index(): This method builds a dense index on a specific embedding column within a dataset.
  • get_nearest_examples(): This method allows for the efficient retrieval of the nearest examples given a query embedding.

This integration removes the need to manually iterate through embedding matrices, enabling the system to scale to massive datasets while maintaining low latency.

Sources