Hugging Face Transformers 4.45.0 Dynamic Speculative Decoding

Hugging Face and Intel Labs have developed dynamic speculative decoding, a method that accelerates text generation by up to 2.7x depending on the task. This technique is the default operational mode for assisted generation starting with the Transformers 4.45.0 release.

How Dynamic Speculative Decoding Works

Dynamic speculative decoding improves upon standard speculative decoding by optimizing the "speculation lookahead" (SL)—the number of tokens a fast draft model generates before a larger target model verifies them in parallel.

While previous methods used a static SL or a heuristic based on the acceptance rate of the previous iteration, dynamic speculative decoding uses the assistant model's own confidence to decide when to stop. Specifically, the system monitors the softmax of the logits for each predicted token. If the assistant model's confidence falls below a predefined assistant_confidence_threshold, token generation halts for that iteration and the sequence is sent to the target model for verification, even if the maximum num_assistant_tokens has not been reached.

Performance Benchmarks

Benchmarking conducted on an RTX 4090 using greedy decoding (temperature = 0) shows that the dynamic approach consistently outperforms heuristic-based methods across various model pairings and tasks:

Target model Draft (Assistant) model Task Speedup - heuristic Speedup - dynamic
facebook/opt-6.7b facebook/opt-125m summarization 1.82x 2.71x
facebook/opt-6.7b facebook/opt-125m open-ended generation 1.23x 1.59x
Salesforce/codegen-6B-mono Salesforce/codegen-350M-mono code generation (python) 0.89x 1.09x
google/flan-t5-xl google/flan-t5-small summarization 1.18x 1.31x
meta-llama/Llama-3.1-8B meta-llama/Llama-3.2-1B summarization 1.00x 1.52x
meta-llama/Llama-3.1-8B meta-llama/Llama-3.2-1B open-ended generation 1.00x 1.18x
meta-llama/Llama-3.1-8B meta-llama/Llama-3.2-1B code generation (python) 1.09x 1.15x

Key findings from these benchmarks include:

  • Llama 3.1/3.2 pairing: The dynamic approach achieved a 1.52x speedup for summarization, whereas the heuristic approach showed no significant speedup.
  • Code generation: For codegen-6B-mono, the heuristic approach actually caused a slowdown (0.89x), while the dynamic approach provided a speedup (1.09x).

Implementation in Transformers 4.45.0

Dynamic speculation is the default mode for assisted decoding in Transformers 4.45.0. It can be implemented by passing an assistant_model to the generate method:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

prompt = "Alice and Bob"
checkpoint = "EleutherAI/pythia-1.4b-deduped"
assistant_checkpoint = "EleutherAI/pythia-160m-deduped"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
inputs = tokenizer(prompt, return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint).to(device)

outputs = model.generate(**inputs, assistant_model=assistant_model)

Configuration Parameters

Users can tune the following parameters in assistant_model.generation_config to optimize performance for specific datasets:

  • assistant_confidence_threshold: The confidence level (softmax of logits) below which the draft model stops generating.
  • num_assistant_tokens: The maximum number of tokens the assistant can generate per iteration.
  • num_assistant_tokens_schedule: Can be set to 'dynamic' (default), 'heuristic', or 'constant' to switch between different speculation strategies.

Theoretical Basis: The Oracle Model

To validate the need for dynamic adjustment, the researchers used an "oracle" that identifies the optimal speculation lookahead for every iteration by generating tokens until a discrepancy occurs between the draft and target models. Analysis of the MBPP dataset and the Alpaca dataset revealed high variance in the optimal number of draft tokens, proving that static lookahead values are suboptimal for maximizing inference speed.

Sources