castorini/pyserini

What is Pyserini?

Pyserini is a Python library that makes it easy to run modern information‑retrieval (IR) experiments from Python code. It wraps two powerful back‑ends:

• Anserini – a Java‑based IR toolkit built on Apache Lucene for classic lexical search (e.g., BM25) and for using Lucene to store dense vectors.
• FAISS – Facebook’s library for fast similarity search over dense vector embeddings.

Together they let you retrieve documents with sparse (term‑based) models, dense (embedding‑based) models, or hybrid combinations, all from a single Python package.

---

Why you might care

• Reproducible research – Pyserini ships with ready‑made indexes, query sets, relevance judgments, and evaluation scripts for many standard test collections (MS MARCO, NaturalQuestions, BEIR, etc.).
• First‑stage retrieval – It is designed to be the fast, scalable “first pass” in a multi‑stage ranking pipeline (e.g., retrieve top‑k candidates, then re‑rank with a large language model).
• Easy installation – pip install pyserini pulls in most dependencies (PyTorch, 🤗 Transformers, ONNX Runtime). FAISS is optional and can be added with the appropriate faiss‑cpu or faiss‑gpu package.
• Programmatic and service‑oriented – Use the library directly in Python, or run a REST or MCP server for remote querying.

---

Getting started (installation)

# Basic install – pulls in most required packages
pip install pyserini

# If you need dense‑vector search, install a FAISS variant first
pip install faiss-cpu   # or faiss-gpu

# Optional multimodal (image) support
pip install "pyserini[optional]"

Pyserini runs on Python 3.12 (other versions may work) and requires Java 21 because it calls into Anserini.

---

Core capabilities

Capability	How you use it	Typical models / back‑ends
Lexical search (BM25, query‑likelihood)	searcher.search("my query")	Lucene via Anserini
Learned sparse retrieval (uniCOIL, SPLADE, etc.)	Same API, just load the appropriate index	Lucene + term‑weight vectors
Dense retrieval (DPR, Contriever, BGE, etc.)	Load a Faiss index or a Lucene‑based vector index	FAISS or Lucene HNSW
Hybrid retrieval (dense + sparse fusion)	Combine results programmatically or use built‑in fusion	Both back‑ends
Index building	IndexBuilder classes for BM25, sparse vectors, dense vectors	Java implementation (direct) or embeddable Python wrappers
Evaluation	Scripts that compute MAP, nDCG, Recall@k on standard qrels	Built‑in evaluation utilities
REST / MCP server	Run python -m pyserini.rest to expose HTTP endpoints	Service‑oriented access

---

Typical workflow

1. Pick a corpus – e.g., MS MARCO, NaturalQuestions, BEIR.
2. Download or build an index – pre‑built indexes are available; otherwise use the indexing guides.
3. Choose a retrieval model – lexical (BM25), learned sparse, dense, or hybrid.
4. Run a search – retrieve the top‑k documents.
5. Fetch document text – use the provided fetch utilities.
6. (Optional) Re‑rank – feed the retrieved passages to a large language model for final scoring.

All steps are documented in the docs/ folder and the README links to concrete examples for each major dataset.

---

Who uses Pyserini?

• Academic researchers publishing IR papers (the library is cited in a SIGIR 2021 paper).
• Teams building retrieval‑augmented generation (RAG) pipelines that need a fast first‑stage retriever.
• Anyone needing reproducible baselines on standard IR test collections.

---

Where to learn more

• Paper: Pyserini: A Python Toolkit for Reproducible Information Retrieval Research (SIGIR 2021).
• Docs: https://github.com/castorini/pyserini/tree/master/docs
• Reproduction guides: 2‑click notebooks for MS MARCO, BEIR, Mr. TyDi, etc.
• Community: Issues and discussions on the GitHub repo, plus a mailing list for the Anserini/Pyserini project.

---

In short: Pyserini lets you run state‑of‑the‑art sparse, dense, and hybrid search from Python with minimal setup, while providing the data, indexes, and evaluation tools needed for rigorous IR research and for building practical retrieval‑augmented applications.