antgroup/vsag

What it solves

VSAG is a vector indexing library designed for high-performance similarity search. It specifically addresses the challenge of searching through massive vector sets that are too large to fit in memory, while providing a simplified interface that allows developers to use it without needing deep expertise in indexing algorithms.

How it works

Written in C++, the library implements several advanced indexing algorithms to handle different data types and search requirements:

• SINDI: A specialized index for sparse vector search that significantly outperforms previous state-of-the-art solutions.
• HGraph: A graph-based index for dense vectors designed for high recall and low latency.
• LazyHGraph: An adaptive index that starts as a brute-force search for small datasets and automatically converts to HGraph as the data grows.
• IVF: An inverted file index optimized for large-scale search and batch queries.

The library supports multiple data types (FP32, INT8, BF16, FP16) and various quantization methods (RaBitQ, PQ, SQ4, SQ8) to reduce memory footprint. It is accessible via C++ natively, as well as through Python (pyvsag) and Node.js/TypeScript bindings.

Who it’s for

It is intended for developers building vector databases or search engines that require high-throughput, low-latency similarity search across both sparse and dense vectors.

Highlights

• Multi-language support: Native C++ with bindings for Python and TypeScript.
• High Performance: Claims significant QPS gains over SOTA algorithms like Zilliz and Glass in sparse vector search.
• C-CPU Optimizations: Includes optimizations for x86_64 (AVX-512, AMX) and ARM (Neon, SVE) architectures.
• Extensive Quantization: Supports a wide range of compression techniques to balance memory usage and recall.
• Industry Adoption: Used by projects like OceanBase, TuGraph, and PolarDB.