huggingface/datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

What it solves

DataTrove is designed to handle the processing, filtering, and deduplication of text data at a massive scale, specifically for preparing training data for Large Language Models (LLMs). It addresses the challenges of managing huge workloads by providing a platform-agnostic framework that maintains low memory usage and supports distributed execution.

How it works

The library uses a pipeline-based architecture where data is processed as a series of Document objects (containing text, an ID, and metadata). Users define a pipeline as a list of processing blocks—such as readers, extractors, filters, and writers—which are then run by an executor.

Execution is parallelized by dividing the total workload into tasks (shards of data), which are assigned to workers (compute resources). The framework supports multiple executors for different environments:

  • LocalPipelineExecutor: For single-machine multiprocessing.
  • SlurmPipelineExecutor: For Slurm clusters using job arrays.
  • RayPipelineExecutor: For Ray clusters.
  • JobsPipelineExecutor: For Hugging Face Jobs.

Who it’s for

It is intended for researchers and engineers working with very large-scale text datasets who need to robustly process and clean data for AI model training.

Highlights

  • Platform-Agnostic: The same pipeline can run locally or on a cluster without modification.
  • Fault Tolerance: Tracks completed tasks via marker files, allowing jobs to be resumed from where they left off after a failure.
  • Fsspec Integration: Supports local, remote, and cloud file systems (e.g., S3, Hugging Face Hub).
  • Prebuilt Blocks: Includes ready-to-use tools for text extraction, tokenization, minhash deduplication, and synthetic data generation.

Related

  • Project
    facebookresearch/stopes

    A library for preparing machine translation data, providing tools for cleaning monolingual data, mining bitext, and running knowledge distillation pipelines.

  • Project
    OpenDCAI/DataFlow

    A data-centric AI system for generating, cleaning, and preparing high-quality LLM training datasets from noisy sources using visual, low-code pipelines.

  • Project
    ZHZisZZ/dllm

    A library that unifies the training, inference, and evaluation of diffusion language models, providing scalable pipelines and recipes for converting AR models into diffusion models.

  • Project
    modelscope/ms-swift

    A large model and multimodal fine-tuning and deployment framework that supports over 1,000 models and provides a full pipeline from training to deployment.

  • Project
    axolotl-ai-cloud/axolotl

    A free and open-source LLM fine-tuning framework that streamlines post-training for text, vision, and audio models using a single YAML configuration.