Porting fairseq WMT19 translation system to 🤗 Transformers

TL;DR

Hugging Face released a port of the fairseq WMT19 translation system to the Transformers library, providing ready‑to‑use models for English‑Russian and English‑German translation that can be loaded with AutoTokenizer and AutoModelForSeq2SeqLM.

Preparations and File Layout

The porting effort began by setting up a working directory and installing the required repositories: fairseq, mosesdecoder, fastBPE, and the transformers library with dev extras. The author created a ~/porting folder, cloned each repo, and installed them in editable mode. The fairseq WMT19 model provides four checkpoints (model1.pt–model4.pt), source and target dictionaries (dict.en.txt, dict.ru.txt), and a BPE codes file (bpecodes). These files were examined to understand the model’s checkpoint, vocabulary, and tokenization artifacts.

Tokenizer Porting

The tokenizer encoder was ported by adapting the existing tokenization_xlm.py file. The author copied tokenization_xlm.py to tokenization_fsmt.py, renamed classes from XLM to FSMT, and removed unused code. Because the WMT19 models use separate source and target vocabularies, the tokenizer’s get_vocab and vocab_size properties were overridden to return the source vocabulary. The BPE handling was changed from the fastBPE style (@@ for non‑final subwords) to the Transformers style ( for final subwords), and the vocabulary remapping was performed using fairseq.data.dictionary.Dictionary.load to obtain the correct id mapping. The decoder was later completed by converting output ids to strings, stripping BPE markers, and applying Moses detokenization.

Model Conversion and Architecture

The conversion script convert_fsmt_original_pytorch_checkpoint_to_pytorch.py was created by starting from the BART conversion script and gradually adding the needed parts. Model weights were extracted from the fairseq checkpoint using the fairseq hub API, which also handles the conversion from the older combined in_proj weights to separate k/q/v projections. Configuration arguments were mapped from fairseq args to Transformers FSMTConfig, including activation_dropout, attention_dropout, d_model, dropout, max_position_embeddings, num_hidden_layers, src_vocab_size, tgt_vocab_size, and token IDs for bos, pad, and eos. The model architecture was derived from modeling_bart.py, with layers adjusted to match fairseq’s TransformerEncoder and TransformerDecoder (e.g., removing unused layers, adding missing ones, and ensuring the correct use of source vs. target vocabulary sizes). The sinusoidal positional embedding was reimplemented as a normal nn.Embedding subclass to satisfy TorchScript requirements while preventing the deterministic weights from being saved.

Testing and Evaluation

Unit tests were added for the tokenizer and modeling components, based on the existing BART test suite but adapted for the dual‑vocabulary setup. A tiny model with random weights was generated for fast CI testing. Manual validation scripts compared the outputs of the fairseq and Transformers implementations token‑by‑token and sentence‑by‑sentence, using a debugger to align intermediate results. Beam search behavior was tuned: the ported model uses early_stopping=False, which was found to give higher BLEU scores than the fairseq default of early_stopping=True when using a beam size of 5. Evaluation on the WMT19 test set with sacrebleu yielded a BLEU score of 39.0498 for the ru‑en direction using beam size 5 and length penalty 1.1. The author noted that the original fairseq paper reports higher scores because it uses an ensemble of four checkpoints and a re‑ranking step, which are not reproduced in the port.

Uploading, Integration and Automation

After conversion, the model files were uploaded to Hugging Face S3 under the author’s account and later moved to the facebook and allenai organizations. The models can be loaded with the standard API, e.g., FSMTTokenizer.from_pretrained("facebook/wmt19-en-ru)). AutoConfig, AutoTokenizer, and AutoModelWithLMHead were updated to recognize the fsmt model type, enabling the pipeline‑style usage. Model cards were written for each variant, detailing language pairs, license, datasets, and evaluation metrics. Documentation was added by adapting the existing BART documentation to FSMT, and the build process was verified with make docs.

Implications and Closing Thoughts

Porting the WMT19 system to Transformers reduced the download size from roughly 13 GB (including optimizer states) to about 1.1 GB per model, making the translators more accessible for downstream use. Although the port does not support the original ensemble of four checkpoints, the single‑checkpoint models still achieve strong translation quality. The effort demonstrated how existing Transformers components (BART‑based modeling, XLM‑based tokenization, conversion utilities) can be reused and adapted to bring high‑quality fairseq models into the library, with the author acknowledging mentorship from Sam Shleifer and contributions from Lysandre Debut and Sylvain Gugger during the PR review process.