mjpost/sacrebleu

What it solves

SacreBLEU addresses the inconsistency and lack of reproducibility in computing BLEU scores for machine translation. It eliminates the "mess" caused by different implementations, varying tokenization methods, and the manual effort required to manage and download standard test sets.

How it works

It wraps the original BLEU reference implementation and provides a standardized way to compute scores on detokenized outputs. The tool handles the downloading and processing of common WMT test sets automatically. To ensure reproducibility, it outputs a unique version signature for every score, allowing researchers to communicate exactly how the score was calculated.

Who it’s for

Researchers and developers working on machine translation and natural language processing who need to shareable, comparable, and reproducible evaluation metrics.

Highlights

• Standardized Metrics: Computes BLEU, chrF, chrF++, and Translation Error Rate (TER).
• Automated Data Management: Automatically downloads and processes common WMT test sets.
• Language Support: Includes specialized tokenizers for Chinese, Japanese (via MeCab), and Korean (via MeCab-ko), as well as a multilingual SentencePiece tokenizer.
• Reproducibility: Generates a detailed version signature for each result to facilitate cross-paper comparisons.
• Statistical Analysis: Supports paired bootstrap resampling and approximate randomization tests for reporting statistical significance.
• Flexible Output: Provides results in JSON, text, or LaTeX table formats.