facebookresearch/faiss

A library for efficient similarity search and clustering of dense vectors.

What it solves

Faiss enables the efficient search and clustering of dense vectors, allowing users to find similar items in datasets of any size, including those that exceed the available RAM. It addresses the problem of high-dimensional vector similarity search, which is computationally expensive when scaled to billions of vectors.

How it works

Faiss uses an index-based system where vectors are stored and compared using L2 (Euclidean) distance or dot products (including cosine similarity). To balance search speed, accuracy, and memory usage, it offers various indexing structures:

  • Exact Search: Simple baselines for high precision.
  • Compressed Representations: Uses binary vectors and compact quantization codes to store vectors in a compressed format, allowing billions of vectors to fit in main memory on a single server.
  • Graph-based Indexes: Implements HNSW and NSG to add indexing structures on top of raw vectors for faster retrieval.
  • Hardware Acceleration: Written in C++ with Python wrappers, it provides high-performance GPU implementations for exact and approximate nearest neighbor search, k-means clustering, and k-selection algorithms.

Who it’s for

Developers and researchers working with high-dimensional embeddings, large-scale similarity search, and clustering tasks in AI and machine learning.

Highlights

  • Billion-scale scalability: Capable of handling datasets that do not fit in RAM using compressed representations.
  • GPU acceleration: Supports single and multi-GPU setups for significantly faster search and clustering.
  • C++ and Python support: Core logic is implemented in C++, providing high-performance efficiency with the Python/numpy interface.
  • Flexible indexing: Offers a variety of trade-offs between search time, quality, and memory usage.

Related

  • Project
    NVIDIA/cuvs

    A GPU-accelerated library for vector search and clustering, providing high-performance approximate nearest neighbors implementations for semantic search and data mining.

  • Project
    unum-cloud/USearch

    A compact, high-performance similarity search and clustering engine for vectors that is faster and more portable than FAISS.

  • Project
    antgroup/vsag

    A C++ vector indexing library for high-performance similarity search, featuring specialized indexes for both sparse and dense vectors and multi-language bindings.

  • Project
    milvus-io/milvus

    Milvus is a high-performance, distributed vector database built for scaling AI applications that require efficient search and organization of massive amounts of unstructured data.

  • Dispatch
    Image Similarity with Hugging Face Datasets and Transformers

    Hugging Face demonstrates how to build an image similarity system using the Transformers and Datasets libraries by computing dense vector embeddings and measuring cosine similarity.