castorini/rank_llm

RankLLM is a Python toolkit for reproducible information retrieval research using rerankers, with a focus on listwise reranking.

What it solves

Search engines often return an initial set of results that are not perfectly ordered by relevance—for example, when using BM25 or similar retrievers. RankLLM reorders (reranks) these candidate lists using large language models, so the most relevant documents appear earlier. It supports three reranking strategies: pointwise (score each document independently, e.g., MonoT5), pairwise (compare pairs of documents, e.g., DuoT5), and listwise (consider the whole list together, e.g., RankZephyr, RankGPT, LiT5). This improves search quality without needing to rebuild the retrieval system.

How it works

The typical workflow is: first, retrieve an initial set of candidates (e.g., top-100 documents using BM25 or SPLADE). Then, feed these candidates to a reranking model, which reorders them based on relevance to the query. RankLLM provides a unified command-line interface (rank-llm) and a Python API. It supports multiple inference backends: local models via vLLM (e.g., RankZephyr, RankVicuna), hosted models via OpenAI/OpenRouter (e.g., GPT-4o), and Gemini (e.g., Gemini 3 Flash). The library also includes evaluation tools (e.g., nDCG@10, MAP, recall), an analysis module for inspecting model invocations, and a data writer to save results in JSONL, TREC, or JSON format. For easy deployment, it can serve reranking over HTTP or via the Model Context Protocol (MCP). Custom prompt templates can be supplied as YAML files.

Who it's for

Researchers and developers working on information retrieval, search quality, and ranking. It is useful for running experiments, reproducing published results (via two-click reproduction commands for MS MARCO datasets DL19–23), and building production systems that need LLM-based reranking. The library is designed to be flexible, with both a CLI for quick runs and a Python API for deeper integration.

Highlights

  • Supports pointwise (MonoT5), pairwise (DuoT5), and listwise (RankZephyr, RankVicuna, RankGPT, LiT5) reranking models
  • Includes a model zoo of pre-trained rerankers (RankZephyr 7B, RankVicuna, LiT5 suite, MonoT5, DuoT5, MonoELECTRA)
  • Works with both open-source models (via vLLM) and proprietary APIs (OpenAI, OpenRouter, Gemini)
  • Can rerank using only the first-token logits (e.g., FirstMistral) for improved inference efficiency
  • Custom prompt templates via YAML files
  • End-to-end pipeline: retrieval, reranking, evaluation, and analysis
  • Two-click reproduction commands for MS MARCO passage datasets (DL19–DL23)
  • Serving options: HTTP API and MCP (Model Context Protocol) server
  • Training scripts for finetuning listwise rerankers
  • Tight integration with popular toolkits such as LlamaIndex (note: README indicates implementation in many toolkits)

Related

  • Project
    FlagOpen/FlagEmbedding

    A comprehensive retrieval toolkit for search and RAG that provides state-of-the-art embedding models and rerankers for text and multimodal data.

  • Project
    CHIANGEL/Awesome-LLM-for-RecSys

    A curated collection of academic papers and resources exploring the integration of Large Language Models into recommender systems, categorized by their role in the recommendation pipeline.

  • Project
    mbzuai-oryx/Awesome-LLM-Post-training

    A curated collection of research papers, benchmarks, and methodologies focused on post-training techniques for Large Language Models, specifically targeting reasoning and decision-making capabilities.

  • Project
    drmingler/smart-llm-loader

    A Python package that converts documents and scanned images into clean, structured markdown chunks using OCR and LLM-powered semantic chunking for RAG systems.

  • Project
    DataScienceUIBK/Rankify

    A comprehensive Python toolkit for retrieval, re-ranking, and RAG, providing a unified interface to experiment with and deploy state-of-the-art models.