NGT-labs/NGT

Nearest Neighbor Search with Neighborhood Graph and Tree for High-dimensional Data

What it solves

NGT addresses the challenge of performing high-speed approximate nearest neighbor (ANN) searches across massive volumes of high-dimensional vector data, which typically ranges from several ten to several thousand dimensions.

How it works

The project provides three primary indexing methods to balance speed, memory, and scale:

  • NGT: A hybrid approach combining graph and tree-based indexing.
  • QG (Quantized Graph): A quantized graph-based method designed for higher performance than the standard NGT approach.
  • QBG (Quantized Blob Graph): A specialized method capable of handling billions of objects.

To optimize resource usage, NGT supports shared memory (via memory-mapped files) to handle datasets larger than available RAM and provides various quantization options (scalar and product quantization) to reduce the data footprint.

Who it’s for

It is designed for developers and researchers working with high-dimensional vector spaces who need a fast, scalable way to retrieve similar items from a large dataset.

Highlights

  • Broad Distance Support: Supports L1, L2, Cosine similarity, Angular, Hamming, Jaccard, Poincare, Lorentz, and Inner product.
  • Massive Scalability: QBG can handle billions of objects.
  • Multi-Language Bindings: Available for Python, Ruby, PHP, Rust, Go, C, and C++.
  • Memory Efficiency: Supports shared memory for indices that exceed physical memory limits.

Related

  • Project
    microsoft/SPTAG

    A library for fast, large-scale approximate nearest neighbor search using space partition trees and relative neighborhood graphs.

  • Project
    datastax/jvector

    A graph-based approximate nearest neighbor search index that combines HNSW and DiskANN to enable fast, memory-efficient vector search over massive datasets.

  • Project
    facebookresearch/faiss

    A library for efficient similarity search and clustering of dense vectors, capable of handling billion-scale datasets using CPU and GPU acceleration.

  • Project
    lmcinnes/pynndescent

    A Python library for fast approximate nearest neighbor search using the Nearest Neighbor Descent algorithm, supporting a wide range of distance metrics.

  • Project
    VexDB-THU/VexDB-Lite

    A high-performance vector similarity search engine for PostgreSQL, DuckDB, and SQLite that uses a custom graph index and quantization to enable efficient ANN search.