microsoft/MS-MARCO-Web-Search
A large-scale information-rich web dataset, featuring millions of real clicked query-document labels
What it solves
Researchers and engineers need realistic, large-scale data to train and test web search retrieval systems. Most public datasets are small or synthetic, so methods that work on them may fail on the web. This project provides a dataset that mirrors real-world web queries and documents, plus standard benchmark tasks to compare different retrieval approaches.
How it works
The dataset is built on ClueWeb22, a collection of about 10 billion web pages. It adds 10 million unique queries from 93 languages, with millions of real click labels collected from Microsoft Bing search logs. For the benchmark, there is a subset of 100 million documents with query–document relevance pairs. Three tasks are defined:
- Embedding model ranking: rank documents given a query using an embedding model, judged on metrics like MRR and recall.
- Embedding retrieval ranking: test approximate nearest neighbor (ANN) search algorithms on precomputed embedding vectors, measuring recall and speed.
- End-to-end retrieval ranking: compare any complete retrieval pipeline (e.g., BM25, embedding + ANN, hybrid, neural indexer) on quality and latency.
Baseline results for each task are provided so new methods can be compared directly.
Who it’s for
This is aimed at researchers and developers working on information retrieval, machine learning, and web search systems. It is also useful for anyone studying embedding models, ANN search, or large-scale data processing. The dataset is meant for non-commercial research only.
Highlights
- 10 billion pages from ClueWeb22 as the document corpus
- 10 million queries in 93 languages with real click labels
- 100 million document subset with train/dev/test splits for benchmarking
- Three complementary retrieval challenges: embedding model, embedding retrieval, and end-to-end
- Baseline numbers from methods like DPR, ANCE, SimANS, SPANN, DiskANN, and BM25
- Additional 5x larger click labels for the full 10 billion document set
Related
- Project
- Project
- Project
- Project