allenai/dolma

What it solves

Dolma provides a high-performance way to curate large-scale datasets for pre-training machine learning models. It addresses the challenge of processing and cleaning billions of documents from diverse sources like web content, academic papers, and code to create a high-quality training corpus.

How it works

The project consists of two parts: a massive 3-trillion-token open dataset and a toolkit. The toolkit uses built-in parallelism to process documents concurrently across single machines, clusters, or cloud environments. It includes pre-built taggers based on popular curation methods (like Gopher and C4) and utilizes a Rust-based Bloom filter for fast document deduplication.

Who it’s for

Researchers and developers building large language models who need tools to curate, clean, and deduplicate massive amounts of raw data for pre-training.

Highlights

• Massive Open Dataset: Access to 3 trillion tokens of diverse web, academic, and code data.
• High Performance: Capable of processing billions of documents concurrently.
• Cloud Native: Supports AWS S3-compatible storage and works across various compute environments.
• Ready-to-use Taggers: Includes standard curation taggers for immediate use.
• Fast Deduplication: Uses Rust for high-speed document removal of duplicates.