hplt-project/sacremoses

What it solves

Sacremoses provides a standardized way to preprocess text for natural language processing (NLP) tasks. It solves the problem of inconsistent text formatting by providing tools to tokenize, detokenize, normalize, and "truecase" text, ensuring that data is clean and consistent before being fed into machine learning models.

How it works

The library offers both a Python API and a Command Line Interface (CLI) to perform several text processing operations:

• Tokenization and Detokenization: Splits text into individual tokens (words, punctuation) and can reverse the process to reconstruct original sentences.
• Truecasing: A process that converts text to its most likely case (e.g., converting "THE ADVENTURES OF SHERLOCK HOLMES" to "the adventures of Sherlock Holmes") based on a trained model.
• Normalization: Cleans text by handling punctuation, removing control characters, and normalizing numbers or quotes.
• Pipelines: The CLI allows users to chain these commands together (e.g., normalize $\rightarrow$ tokenize $\rightarrow$ truecase) in a single execution pipeline.

Who it’s for

It is designed for NLP researchers and developers who need to prepare large datasets for training or evaluating machine translation and other text-based AI models.

Highlights

• Comprehensive Toolset: Includes tokenization, detokenization, truecasing, and normalization in one package.
• CLI Pipeline Support: Ability to chain multiple preprocessing steps via the command line for efficient data processing.
• Trainable Truecaser: Includes the ability to train a custom truecase model from a text file.
• Multiprocessing: Supports the -j flag in the CLI to speed up processing using multiple CPU cores.