zilliztech/VectorDBBench

VectorDBBench (VDBBench) – A Benchmark Suite for Vector Databases

What it is – VDBBench is a Python‑based tool that lets you run repeatable performance and cost‑effectiveness tests on a wide range of vector‑search back‑ends (Milvus, Qdrant, Pinecone, Weaviate, Elastic, PostgreSQL‑based solutions, Redis, etc.). It ships with a small visual UI for launching benchmarks and comparing results, and it can also be driven entirely from the command line.

Why it matters – Choosing a vector database (or a cloud‑hosted vector‑search service) is often a trial‑and‑error process. VDBBench provides a single, consistent workload definition (insertion, nearest‑neighbor search, filtered search, and now BM25‑style full‑text search) together with a catalog of public datasets (SIFT, GIST, Cohere Wikipedia embeddings, OpenAI‑generated C4 embeddings, and multimodal VIBE datasets hosted on Hugging Face). This lets practitioners compare latency, throughput (QPS), recall, and even dollar‑per‑query costs across providers.

Key features (as described in the README)

• Supported back‑ends – Milvus/Zilliz Cloud, Qdrant, Pinecone, Weaviate, Elastic (including Elastic Cloud), OpenSearch/AWS OpenSearch Serverless, PostgreSQL extensions (pgvector, pgvecto.rs, VectorChord, etc.), Redis, MongoDB, TiDB, Vespa, and many more via optional extras.
• Workloads – Insertion, vector similarity search, filtered search, and full‑text BM25 retrieval (MS MARCO, HotpotQA). Each workload can be run in performance mode (large‑scale, high‑concurrency) or capacity mode (varying dimensions and dataset sizes).
• Dataset catalog – Classic low‑dimensional benchmarks (SIFT, GIST) plus modern high‑dimensional embeddings (200‑dim cosine, 768‑dim, 1536‑dim) and multimodal Parquet/HDF5 collections ranging from 1 M to 100 M vectors.
• Cost‑effectiveness reporting – For cloud services the tool can compute cost per query based on the provider’s pricing model, helping users pick the cheapest option that meets latency/recall targets.
• Extensible CLI – vectordbbench <backend> [options] where each backend has its own set of flags (index parameters, concurrency, authentication, etc.). A --note/--note-file flag lets you embed deployment context into the JSON result files.
• Visual UI – An optional web interface (not detailed in the README) that displays benchmark runs and comparative charts, making it easier for non‑experts to reproduce results.
• Installation – Single‑command pip install, with optional extras for the specific database client, e.g. pip install 'vectordb-bench[pinecone]'.

Typical usage

# Install core package (includes Milvus client)
pip install vectordb-bench

# Install extra for a specific backend, e.g. Qdrant
pip install 'vectordb-bench[qdrant]'

# Run a benchmark against a local Milvus instance
vectordbbench milvushnsw \
  --uri http://localhost:19530 \
  --collection-name VDBBench \
  --case-type Performance768D1M \
  --m 16 --ef-construction 128 --ef-search 128 \
  --k 10 --nq 2 \
  --skip-drop-old --skip-load

The command prints a JSON result file containing throughput (qps), latency (conc_latency_*), recall, and any user‑provided notes.

Who should use it –

• Data‑science teams evaluating which vector store to adopt for a new similarity‑search product.
• Cloud‑cost engineers needing a quantitative comparison of hosted vector‑search services.
• Researchers benchmarking new indexing algorithms against a common set of public datasets.
• Anyone who wants a reproducible, scriptable way to measure insertion speed, query latency, and recall across many vector‑DB implementations.

Where to find more – The package is published on PyPI (vectordb-bench), the source lives under the zilliztech/VectorDBBench GitHub repo, and a live leaderboard is hosted at https://zilliz.com/benchmark. Documentation for each release (e.g., full‑text search, Hugging Face dataset integration) is linked in the repo’s docs/ folder.