harsha-simhadri/big-ann-benchmarks

Framework for evaluating ANNS algorithms on billion scale datasets.

What it solves

Finding the closest items in huge datasets (billions of entries) is a core task in search and recommendation systems. Different algorithms and hardware perform very differently at that scale, and there was no standard way to compare them fairly. This project gives researchers and engineers a common set of benchmarks to measure and compare practical nearest-neighbor search approaches.

How it works

It provides standardized datasets and evaluation procedures. The project is a fork of the ann-benchmarks project, extended to focus on billion-scale datasets and practical variants of nearest-neighbor search. It includes separate guides for two NeurIPS competitions: the 2021 "Billion-Scale ANN" competition and the 2023 "Practical Vector Search" competition. Each competition has its own README explaining how to run the benchmarks and summarize results. Submissions can be made to the NeurIPS 2023 ongoing leaderboard by following the instructions in the corresponding README.

Who it’s for

Researchers and engineers working on approximate nearest neighbor search, vector databases, and large-scale information retrieval. It is also useful for teams evaluating commercial or open-source search systems against a public standard.

Highlights

  • Targets billion-scale datasets, unlike earlier smaller benchmarks
  • Covers both algorithmic and hardware performance
  • Supports ongoing leaderboard submissions for the NeurIPS 2023 competition
  • Based on the widely used ann-benchmarks project, adding new practical variants

Related

  • Project
  • Project
  • Project
  • Project