facebookresearch/stopes

A library for preparing data for machine translation research (monolingual preprocessing, bitext mining, etc.) built by the FAIR NLLB team.

What it solves

stopes is a library designed to simplify the preparation of data for machine translation research. It addresses the challenges of processing large-scale web data into high-quality training sets, specifically for the No Language Left Behind (NLLB) project.

How it works

The library is organized into several functional components:

  • Core: A framework for writing readable and scalable data pipelines.
  • Modules: Pre-built steps for common mining and evaluation tasks.
  • Pipelines: Implementation of specific workflows, including:
    • Monolingual: Cleaning and preprocessing single-language data.
    • Bitext: Extracting aligned sentence pairs from two monolingual datasets (global mining).
    • Distillation: Training a small student model using a pre-trained large teacher model via sequence-level knowledge distillation.
  • Eval: Tools for evaluation, including text-free speech translation evaluation (ALTI+ and BLASER).
  • Demo: Practical examples, such as mining demos and toxicity/hallucination analysis.

Who it’s for

Researchers and developers working on machine translation, specifically those needing to process massive amounts of web data to create clean training sets or perform knowledge distillation.

Highlights

  • Scalable Pipelines: Built to handle large-scale data processing for multilingual translation.
  • Bitext Mining: Includes tools to extract aligned sentences from monolingual data.
  • Knowledge Distillation: Supports sequence-level distillation to create efficient student models.
  • Speech Translation Eval: Provides specialized tools like ALTI+ and BLASER for evaluating speech translation without needing text references.

Related

  • Project
    huggingface/datatrove

    A library for processing, filtering, and deduplicating massive text datasets at scale, specifically optimized for LLM training data preparation.

  • Project
    modelscope/easydistill

    A config-driven knowledge distillation toolkit that turns black-box teacher models into high-quality SFT and DPO training data across text, image, and video modalities.

  • Project
    ZHZisZZ/dllm

    A library that unifies the training, inference, and evaluation of diffusion language models, providing scalable pipelines and recipes for converting AR models into diffusion models.

  • Project
    lhotse-speech/lhotse

    A Python library for flexible multimodal data preparation that optimizes the loading and manipulation of speech, audio, video, and image data for ML training.

  • Project
    IBM/lale

    A Python library for semi-automated data science that automates algorithm selection and hyperparameter tuning for scikit-learn compatible pipelines.