Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models – Hugging Face Blog Summary

Leveraging Pre-trained Checkpoints for Encoder-Decoder Models

The Hugging Face blog post from November 9 2020 describes how to initialize encoder‑decoder (seq2seq) models with weights from pre‑trained encoder‑only or decoder‑only checkpoints such as BERT, RoBERTa, or GPT2. This warm‑starting technique avoids the costly pre‑training of a full encoder‑decoder model and yields results competitive with models like T5 and Pegasus on a range of sequence‑to‑sequence tasks.

Theory of Warm‑starting

An encoder‑decoder model consists of an encoder stack and a decoder stack. Warm‑starting can be done in four ways: (1) both encoder and decoder from an encoder‑only checkpoint (e.g., BERT), (2) encoder from an encoder‑only checkpoint and decoder from a decoder‑only checkpoint (e.g., BERT + GPT2), (3) only the encoder from an encoder‑only checkpoint, or (4) only the decoder from a decoder‑only checkpoint. When warming‑starting from BERT, the encoder layers map one‑to‑one to BERT layers and are initialized with BERT weights. The decoder receives the same self‑attention and LM‑Head weights from BERT, but cross‑attention layers are added and randomly initialized because BERT has no cross‑attention. When warming‑starting the decoder from GPT2, the self‑attention and LM‑Head can be copied directly, while cross‑attention layers are again randomly initialized.

If the encoder and decoder architectures are identical (excluding cross‑attention), their weights can be tied, halving the parameter count. This weight‑tying is only meaningful when both sides are warmed‑started from the same encoder‑only checkpoint.

Analysis of Warm‑started Models

The blog summarizes the experiments from Rothe et al. (2020) which compared a variety of warm‑started encoder‑decoder configurations against randomly initialized baselines on four task groups: sentence fusion, sentence splitting, machine translation (WMT14 EN↔DE), and abstractive summarization (CNN/Dailymail, BBC XSum, Gigaword). All models used a 12‑layer, 768‑dim hidden size corresponding to bert‑base‑cased, roberta‑base, or gpt2 checkpoints.

The table of model variants shows the number of randomly initialized (“random”) and leveraged (“leveraged”) parameters. For example:

  • Rnd2Rnd: 221 M random, 0 leveraged
  • Rnd2BERT / BERT2Rnd: 112 M random, 109 M leveraged
  • BERT2BERT: 26 M random, 195 M leveraged
  • BERTShare / RoBERTaShare: 26 M random, 109 M / 126 M leveraged (due to weight tying)
  • BERT2GPT2: 26 M random, 234 M leveraged
  • RoBERTa2GPT2: 26 M random, 250 M leveraged

Results (quoted from the tables) indicate:

  • Sentence Fusion (DiscoFuse, SARI): RoBERTa2GPT2 achieved 89.9 on 100 % data and 87.1 on 10 % data; RoBERTaShare (large) reached 90.3 / 87.7.
  • Sentence Splitting (WikiSplit, SARI): BERTShare scored 63.5, RoBERTaShare 63.4, RoBERTaShare (large) 63.8.
  • Machine Translation (WMT14, BLEU‑4): BERT2Rnd and BERT2BERT both reached 30.1 → 32.7 (EN→DE / DE→EN). BERT2Rnd (large, custom) improved to 31.7 → 34.2. GPT2‑based models performed poorly on EN→DE (e.g., BERT2GPT2 23.2) because GPT2’s vocabulary is English‑only.
  • Summarization (Rouge‑2): RoBERTaShare achieved 18.95 on CNN/Dailymail, 17.50 on BBC XSum, 19.70 on Gigaword. RoBERTaShare (large) reached 18.91 on CNN/Dailymail, 18.79 on BBC XSum, 19.78 on Gigaword. BERTShare and BERT2BERT were close behind, while GPT2‑based models lagged (e.g., BERT2GPT2 4.96 on CNN/Dailymail).

The analysis concludes that warming‑starting the encoder gives a consistent boost across tasks, whereas warming‑starting the decoder adds less benefit because cross‑attention layers remain randomly initialized. Weight sharing helps when input and output distributions are similar (e.g., BBC XSum) but can hurt translation where model capacity and divergent vocabularies matter. Matching the checkpoint vocabulary to the task language is essential.

Practice: Warm‑starting with 🤗Transformers

The blog provides a full notebook that warm‑starts a BERT2BERT model and fine‑tunes it on CNN/Dailymail summarization.

  1. Install libraries: datasets==1.0.2 and transformers==4.2.1.
  2. Load and preprocess data: Tokenize articles with bert‑base‑uncased (max length 512) and highlights (max length 128), replace padding label tokens with ‑100.
  3. Warm‑start the model:
    from transformers import EncoderDecoderModel
bert2bert = EncoderDecoderModel.from_encoder_decoder_pretrained(
    "bert-base-uncased", "bert-base-uncased"
)
    The warning shows that the classifier (cls) weights are unused and cross‑attention weights are newly initialized, as expected.
  4. Set generation parameters (copied from bart‑large‑cnn):
    bert2bert.config.decoder_start_token_id = tokenizer.cls_token_id
bert2bert.config.eos_token_id = tokenizer.sep_token_id
bert2bert.config.pad_token_id = tokenizer.pad_token_id
bert2bert.config.vocab_size = bert2bert.config.encoder.vocab_size
bert2bert.config.max_length = 142
bert2bert.config.min_length = 56
bert2bert.config.no_repeat_ngram_size = 3
bert2bert.config.early_stopping = True
bert2bert.config.length_penalty = 2.0
bert2bert.config.num_beams = 4
  5. Fine‑tune with Seq2SeqTrainer and Seq2SeqTrainingArguments, using predict_with_generate=True and a compute_metrics function that returns Rouge‑2 precision, recall, and fmeasure.
  6. Train on a subset (32 training examples, 8 validation examples) for demonstration; full training on a TITAN RTX takes ~8 hours.
  7. Evaluate on the test set; the fully trained BERT2BERT model (patrickvonplaten/bert2bert_cnn_daily_mail) achieves a Rouge‑2 fmeasure of 18.22 on the complete CNN/Dailymail evaluation, matching or slightly exceeding the numbers reported in the paper.

The notebook also shows how to save and reload the model, and how to tie encoder‑decoder weights by passing tie_encoder_decoder=True to from_encoder_decoder_pretrained.

Takeaway

Warm‑starting encoder‑decoder models with existing BERT, RoBERTa, or GPT2 checkpoints lets practitioners obtain strong seq2seq performance without the prohibitive compute of pre‑training from scratch. Encoder initialization is the most critical factor; decoder initialization adds less value unless cross‑attention is learned during fine‑tuning. Weight tying is beneficial when the input and output languages or formats are similar, and vocabulary match between checkpoint and task language is a prerequisite for success.

Sources