NGT-labs/NGT

Nearest Neighbor Search with Neighborhood Graph and Tree for High-dimensional Data

What it solves

NGT addresses the challenge of performing high-speed approximate nearest neighbor (ANN) searches across massive volumes of high-dimensional vector data, which typically ranges from several tens to several thousand dimensions.

How it works

NGT employs three primary indexing methods to optimize search speed and memory usage:

  • NGT: A hybrid approach combining graph and tree-based indexing.
  • QG (Quantized Graph): A quantized graph-based method designed for higher performance than the standard graph/tree approach.
  • QBG (Quantized Blob Graph): A specialized method capable of handling billions of objects.

To manage datasets that exceed available RAM, the project supports shared memory via memory-mapped files. It also provides various quantization options (Product, Scalar) to further optimize data representation.

Who it’s for

Developers and data engineers who need to implement efficient similarity searches in high-dimensional spaces across various platforms (Linux, macOS) and programming languages including Python, C++, Go, Rust, Ruby, and PHP.

Highlights

  • Broad Distance Support: Supports L1, L2, Cosine similarity, Angular, Hamming, Jaccard, Poincare, Lorentz, and Inner product.
  • Massive Scalability: QBG can handle billions of objects, and shared memory options allow for indexing data larger than system memory.
  • Multi-Language Support: Native libraries and wrappers for a wide array of popular programming languages.
  • Flexible Data Types: Supports 4-byte floats, 2-byte floats (FP16), and 1-byte unsigned integers.

Related

  • Project
    microsoft/SPTAG

    A library for fast, large-scale approximate nearest neighbor search using space partition trees and relative neighborhood graphs.

  • Project
    NVIDIA/cuvs

    A GPU-accelerated library for vector search and clustering, providing high-performance approximate nearest neighbors implementations for AI and data mining.

  • Project
    lmcinnes/pynndescent

    A Python library for fast approximate nearest neighbor search using the Nearest Neighbor Descent algorithm, supporting a wide range of distance metrics.

  • Project
    facebookresearch/faiss

    A library for efficient similarity search and clustering of dense vectors, capable of handling billion-scale datasets using CPU and GPU acceleration.

  • Project
    datastax/jvector

    JVector is a Java library that implements a fast, scalable graph‑based approximate nearest‑neighbor index (combining HNSW and DiskANN ideas). It supports two‑pass searches with compressed and full‑precision vectors, incremental construction, and native SIMD acceleration, enabling large‑than‑memory vector search for AI‑driven semantic search and recommendation systems.