zilliztech/VectorDBBench

Benchmark for vector databases.

What it solves

VectorDBBench (VDBBench) provides a standardized way to compare the performance and cost-effectiveness of various vector databases and cloud services. It eliminates the guesswork for users by allowing them to reproduce benchmark results or test new systems across a wide array of supported backends, helping them find the optimal database for their specific needs.

How it works

The tool mimics real-world production environments by running diverse testing scenarios, including data insertion, searching, and filtered searching. It utilizes public datasets from actual production scenarios (such as SIFT, GIST, and Cohere) and supports a vast number of database clients. Users can initiate benchmarks via an intuitive visual interface or a command-line interface (CLI) with detailed configuration options for indexing parameters, sharding, and concurrency.

Who it’s for

It is designed for developers and system architects who need to evaluate and select a vector database, as well as non-professionals who want to easily reproduce benchmark results.

Highlights

  • Extensive Database Support: Supports a wide range of open-source and cloud vector databases including Milvus, Pinecone, Qdrant, Weaviate, Elastic, pgvector, and many others.
  • Visual Interface: Includes an intuitive UI for initiating benchmarks and viewing comparative reports.
  • Real-world Scenarios: Uses production-grade datasets and tests scenarios like insertion and filtered search.
  • Cost-Effectiveness Reports: Specifically provides cost analysis for cloud services to make benchmarking more realistic.
  • Full Text Search Benchmarking: Recently added support for BM25-style retrieval benchmarking across supported backends.

Related

  • Project
    qdrant/vector-db-benchmark

    A general framework for benchmarking different vector search engines under the same hardware constraints to compare their performance and efficiency.

  • Project
    Vchitect/VBench

    VBench is an open‑source benchmark suite that automatically evaluates video‑generation models across dozens of quality dimensions (technical, aesthetic, and trustworthiness). It includes prompt collections, metric pipelines, human‑aligned scores, and public leaderboards for T2V, I2V, long‑video, and intrinsic‑faithfulness evaluation.

  • Project
    alibaba/zvec

    An open-source, in-process vector database that provides low-latency, scalable similarity search and hybrid search capabilities embedded directly into applications.

  • Project
    sdv-dev/SDGym

    A benchmarking framework for modeling and generating synthetic data, allowing users to compare performance, memory usage, and quality across different ML techniques.

  • Project
    embeddings-benchmark/mteb

    A multimodal toolbox for evaluating embeddings and retrieval systems, providing a standardized benchmark and leaderboard for comparing model performance.