Faster Text Generation with Self-Speculative Decoding

Self-speculative decoding, implemented through the LayerSkip method, enables a single large language model (LLM) to accelerate text generation by using its own early layers to draft tokens and its deeper layers to verify them. This approach reduces the memory footprint and computational latency associated with traditional speculative decoding, which requires two separate models.

Self-Speculative vs. Traditional Speculative Decoding

Traditional speculative decoding relies on a small draft model to generate a sequence of tokens and a larger verification model to validate them. In contrast, self-speculative decoding uses the same model for both roles. By exiting the model at an intermediate layer to produce draft tokens, the system eliminates the need for a secondary model's weights, thereby saving memory and reducing the overall hardware footprint for large-scale inference.

Technical Implementation: Early Exit and Unembedding

To enable self-speculative decoding, the model must be capable of "early exiting," where the generation process stops at a pre-specified intermediate layer.

Unembedding Logits

Because the language model (LM) head is typically trained only for the final layer, the model requires specific training to "unembed" logits from intermediate layers. This process projects the intermediate layer's output onto the LM head to predict the next token.

Training Modifications

Two primary modifications are used during pretraining or fine-tuning to make early exits viable:

  • Layer Dropout: The model is trained to skip certain layers, with dropout rates increasing in deeper layers. This reduces reliance on later layers and improves generalization.
  • Early Exit Loss: A normalized loss is applied to each exit (intermediate layer), forcing the LM head to learn how to unembed outputs from various depths across the model.

The Inference Process: Drafting and Verification

Inference occurs in two distinct stages:

  1. Self-Drafting: Tokens are generated by exiting the model at an intermediate layer. The number of speculative tokens and the specific exit layer are hyperparameters that balance speed and accuracy.
  2. Self-Verification: The full model verifies the draft tokens. To optimize this, the system reuses the results from the early layers cached during the drafting phase, computing only the remaining layers for verification.

Hardware and Performance Optimizations

Self-speculative decoding utilizes a KVQ cache (a combination of the KV cache and an exit query cache) to minimize redundant calculations. This provides three primary advantages over two-model systems:

  • Shared Weights: The first $E$ layers are reused for both drafting and verification.
  • Shared KV Cache: Key-value pairs from the first $E$ layers are reused.
  • Shared Compute: The exit query cache stores the query vector of the exit layer $E-1$, allowing the verification process to skip the computation of layers $0$ to $E-1$.

Benchmarking and Performance Trade-offs

Benchmarks conducted on A100 GPUs show that for most model sizes—including Llama 3.2 1B, Llama 3 8B, Llama 2 13B, and Llama 2 7B—early-exit self-speculative decoding is faster than traditional two-model speculative decoding.

The "Sweet Spot" for Early Exits

There is a non-linear relationship between the exit layer and performance. Exiting too early results in low accuracy and a low acceptance rate for draft tokens, while exiting too late increases the computational overhead of the drafting stage. For most models, there is a "sweet spot" where the trade-off between prediction accuracy and generation overhead is optimized for maximum tokens per second.

Model-Specific Results

  • Llama 2 70B: While significantly faster than autoregressive decoding, it showed more limited speedups compared to other models, potentially due to being continually pretrained with fewer tokens (328M tokens) compared to the Llama 2 7B variant (52B tokens).
  • Baseline Models: Models not trained with the LayerSkip recipe do not see speedups because their early layers are not trained to predict the output, leading to very low token acceptance rates.

Integration with Transformers

Self-speculative decoding is integrated into the Hugging Face transformers library. It can be enabled by adding the assistant_early_exit argument to the generate() function, provided the model checkpoint has been trained with the LayerSkip recipe.

Sources