qdrant/vector-db-benchmark

Framework for benchmarking vector search engines

What it solves

It provides a standardized framework to measure and compare the performance of different vector search engines. Because different engines offer varying features and efficiency, this tool allows users to evaluate them under identical hardware constraints to determine which one best fits their specific needs.

How it works

The project operates in a server-client architecture where the server (the vector database) runs in a Docker container and a Python-based client executes the tests. The process involves:

  1. Server Setup: Launching the engine via Docker Compose.
  2. Client Execution: Using a Python script to run benchmarks against specified engines and datasets.
  3. Configuration: Using dedicated configuration files to define connection, collection creation, data upload, and search parameters.
  4. Extensibility: New engines can be added by implementing base classes for configuration, uploading, and uploading data, and searching.

Who it’s for

Developers and engineers who need to objectively compare the performance, throughput, and efficiency of various vector databases before choosing one for their production environment.

Highlights

  • Standardized Testing: Benchmarks different engines under the same hardware conditions.
  • Docker-based Deployment: Simplifies server setup using Docker Compose.
  • Flexible Configuration: Supports custom parameters for collection creation, indexing, and searching.
  • Extensible Architecture: Provides base classes to easily implement and register new vector search engines.

Related

  • Project
    zilliztech/VectorDBBench

    A benchmark tool for comparing the performance and cost-effectiveness of various open-source and cloud vector databases using real-world production datasets.

  • Project
    alibaba/zvec

    An open-source, in-process vector database that provides low-latency, scalable similarity search and hybrid search capabilities embedded directly into applications.

  • Project
    openml/automlbenchmark

    A framework for evaluating and comparing open-source AutoML systems using standardized environments and curated datasets to ensure reproducible research.

  • Project
    Restream/reindexer

    An embeddable, in-memory document database written in C++ and Go that provides a high-performance alternative to Elasticsearch with support for vector and full-text search.

  • Project
    Cloud-CV/EvalAI

    An open-source platform for evaluating and comparing machine learning and AI algorithms at scale through standardized leaderboards and submission interfaces.