vdaas/vald

Vald. A Highly Scalable Distributed Vector Search Engine

What it solves

Vald is a distributed search engine designed to handle the challenge of searching through billions of dense feature vectors. It provides a highly scalable way to perform approximate nearest neighbor (ANN) searches across massive datasets of vector data, which are often generated from images, audio, video, or text.

How it works

Vald uses a cloud-native architecture implemented on Kubernetes. It leverages the NGT (Neighborhood Graph Tearing) algorithm to achieve fast approximate nearest neighbor searches. The system is composed of several distributed components, including Agent NGT for the search logic, an Index Manager for handling vector indexing and backups, and various Gateways (Load Balancer and Filter) to manage request flow.

Who it’s for

It is intended for developers and organizations building large-scale AI applications that require high-performance vector search, such as recommendation systems, recognition services, and real-time translators.

Highlights

  • Cloud-Native Design: Built specifically for Kubernetes to allow for horizontal scaling.
  • Massive Scale: Capable of searching through billions of feature vectors.
  • Automatic Indexing: Features automatic vector indexing and index backup capabilities.
  • Versatile Data Support: Can be used for any data type (text, image, audio, video) as long as it is converted into a vector.

Related

  • Project
    vearch/vearch

    Vearch is a cloud-native distributed vector database that enables efficient similarity search of embedding vectors for AI applications.

  • Project
    NGT-labs/NGT

    A high-speed library for approximate nearest neighbor searches in high-dimensional vector spaces, supporting graph, tree, and quantized indexing methods.

  • Project
    elastic/elasticsearch

    A distributed search and analytics engine and vector database that enables near real-time search and analysis over massive datasets for AI and monitoring applications.

  • Project
    microsoft/SPTAG

    A library for fast, large-scale approximate nearest neighbor search using space partition trees and relative neighborhood graphs.

  • Project
    qdrant/qdrant

    A high-performance vector database and similarity search engine written in Rust, designed for storing and retrieving embeddings with advanced filtering and hybrid search capabilities.