microsoft/SPTAG

A distributed approximate nearest neighborhood search (ANN) library which provides a high quality vector index build, search and distributed online serving toolkits for large scale vector search scenario.

What it solves

SPTAG is designed for large-scale vector approximate nearest neighbor (ANN) search. It allows users to efficiently find vectors in a massive dataset that are most similar to a given query vector based on L2 or cosine distances.

How it works

The library combines space partition trees and relative neighborhood graphs (RNG) to optimize search. It offers two primary methods:

  • SPTAG-KDT: Uses kd-trees and RNG, which is more efficient during the index building process.
  • SPTAG-BKT: Uses balanced k-means trees and RNG, which provides better search accuracy for very high-dimensional data.

The search process begins by using the space partition trees to find initial seed vectors, followed by iterative searches within the RNG to refine the results.

Who it’s for

Developers and researchers working with high-dimensional vector datasets who need a fast, scalable ANN search library for billion-scale datasets.

Highlights

  • Fresh update: Supports online vector insertion and deletion.
  • Distributed serving: Capable of serving searches across multiple machines.
  • Detailed distance metrics: Supports both L2 and cosine distances.

Related

  • Project
    NGT-labs/NGT

    A high-speed approximate nearest neighbor search library for high-dimensional vector data, supporting graph, tree, and quantized indexing methods.

  • Project
    lmcinnes/pynndescent

    A Python library for fast approximate nearest neighbor search using the Nearest Neighbor Descent algorithm, supporting a wide range of distance metrics.

  • Project
    antgroup/vsag

    A C++ vector indexing library for high-performance similarity search, featuring specialized indexes for both sparse and dense vectors and multi-language bindings.

  • Project
    microsoft/DiskANN

    A composable vector indexing library that enables scalable and cost-effective nearest-neighbor search across various database storage backends.

  • Project
    datastax/jvector

    A graph-based approximate nearest neighbor search index that combines HNSW and DiskANN to enable fast, memory-efficient vector search over massive datasets.