Transformer-based Encoder-Decoder Models Hugging Face Blog Post 2020

TL;DR

Hugging Face published an educational blog post that walks through the transformer-based encoder-decoder model, explaining its encoder and decoder components, how self-attention and cross-attention work, and demonstrating autoregressive generation with the 🤗Transformers library using a MarianMT translation example.

Background

The blog post frames sequence-to-sequence tasks as mapping an input sequence of vectors to a target sequence of unknown length, a formulation that motivated the development of encoder-decoder models.

"Tasks in natural language generation (NLG), a subfield of NLP, are best expressed as sequence-to-sequence problems."

It notes that early RNN-based encoder-decoders could handle variable-length outputs but suffered from vanishing gradients and limited parallelization.

"RNNs suffer from the vanishing gradient problem, making it very difficult to capture long-range dependencies... Second, the inherent recurrent architecture of RNNs prevents efficient parallelization when encoding."

The post then introduces the Transformer architecture from Vaswani et al. (2017) as the solution that enables highly parallelizable processing of variable-length sequences.

Encoder-Decoder Architecture

The transformer-based encoder-decoder consists of an encoder stack and a decoder stack, each built from residual attention blocks.

"Analogous to RNN-based encoder-decoder models, transformer-based encoder-decoder models consist of an encoder and a decoder which are both stacks of residual attention blocks."

The encoder maps the input sequence to a sequence of contextualized hidden states, while the decoder models the conditional distribution of the target sequence given those encodings and previously generated tokens.

"The transformer-based encoder part encodes the input sequence ... to a sequence of hidden states... The transformer-based decoder part then models the conditional probability distribution of the target vector sequence ... given the sequence of encoded hidden states."

Auto‑regressive generation proceeds step‑by‑step, with the decoder re‑using the encoder’s output after the first forward pass.

"It is important to understand that the encoder is only used in the first forward pass to map ... As of the second forward pass, the decoder can directly make use of the previously calculated encoding."

Encoder Details

Each encoder block contains a bi‑directional self‑attention layer followed by two feed‑forward layers. The self‑attention layer projects input vectors to queries, keys, and values, computes attention weights via softmax, and returns a weighted sum of values plus the original input.

"Each input vector ... is projected to a key vector, value vector and query vector... an output vector is defined as the weighted sum of all value vectors ... plus the input vector."

Because every query attends to all keys, the encoder captures long‑range dependencies in a single operation and enables full parallelization across positions.

"The output ... is computed via a series of matrix multiplications and a softmax operation, which can be parallelized effectively."

A code snippet shows that changing the last word of the input alters the encoder’s representation of the first token, confirming context‑dependence.

Decoder Details

Each decoder block contains a uni‑directional self‑attention layer, a cross‑attention layer, and two feed‑forward layers. Uni‑directional self‑attention restricts each query to attend only to its own position and previous positions, ensuring autoregressive behavior.

"In uni‑directional self‑attention, each query vector is compared only to its respective key vector and all previous ones... This prevents an output vector to include any information about the following input vector."

Cross‑attention then conditions the decoder’s representation on the full encoder output by projecting the decoder’s hidden states as queries against the encoder’s keys and values.

"The cross-attention layer puts each of its input vectors into relation with all contextualized encoding vectors to condition the probability distribution of the next target vectors on the input of the encoder as well."

The decoder’s final linear layer (LM head) maps hidden states to logits over the vocabulary, which are turned into probabilities by softmax.

"The 'LM head' maps the encoded sequence of target vectors to a sequence of logit vectors... a probability distribution over the whole vocabulary can be obtained by applying a softmax operation."

A code example demonstrates that altering a decoder token after a given position does not affect the logit for an earlier token, illustrating the causal nature of uni‑directional self‑attention.

Inference with 🤗Transformers

The post shows how to run translation with a pretrained MarianMT model using the 🤗Transformers library, highlighting that the generate() method handles encoding, decoder initialization with a BOS/pad token, and beam‑search decoding internally.

"Calling .generate() does many things under-the-hood. First, it passes the input_ids to the encoder. Second, it passes a pre-defined token, which is the <pad> symbol... along with the encoded input_ids to the decoder. Third, it applies the beam search decoding mechanism...

The provided example translates "I want to buy a car" to German, producing the output <pad> Ich will ein Auto kaufen.

Appendix: Code Snippets

The appendix includes a minimal greedy‑decoding loop that manually steps through encoder and decoder calls, argmax‑selects the next token, and concatenates it to the decoder input, reproducing the first few words of the translation.

"In this code example, we show exactly what was described earlier... As a result, the model has generated the words 'Ich will ein'."

The post concludes that readers now have a detailed understanding of how transformer‑based encoder‑decoder models work and how to use them with 🤗Transformers, while noting that training details will be covered in a future post.