Intel Gaudi Assisted Generation Support
Hugging Face has integrated assisted generation, also known as speculative sampling, into the Optimum Habana library to accelerate text generation on Intel Gaudi processors. This optimization reduces inference latency, infrastructure costs, and power consumption for large-scale Generative AI implementations.
Assisted Generation via Speculative Sampling
Assisted generation speeds up text generation by using a smaller draft model to predict multiple tokens before they are verified by a larger target model. The process follows a specific cycle:
- Drafting: A draft model generates $K$ tokens.
- Evaluation: The target model evaluates these $K$ tokens.
- Correction: If the draft model's tokens are rejected, the target model generates the next correct token.
- Iteration: The process repeats, starting again with the draft model for the next $K$ tokens.
This technique maintains the same sampling quality as standard autoregressive sampling because the target distribution is recovered during the speculative sampling process. The primary factors determining the effectiveness of this method are the relative size of the draft model compared to the target model and the acceptance rate of the draft tokens.
Technical Implementation on Intel Gaudi
Implementing assisted generation on Intel Gaudi requires managing distinct optimization strategies for models of different sizes. Specifically, the implementation leverages KV caching and quantized models, where each model (draft and target) maintains its own separate KV cache to handle their differing sizes.
This functionality is now part of Optimum Habana, which extends Hugging Face libraries such as Transformers and Diffusers to ensure AI workflows are fully optimized for Intel Gaudi hardware.
Performance and Usage
Assisted generation typically provides speedups of approximately 2x for large transformer-based models. The actual acceptance rate of the draft model—and thus the resulting speedup—is partially dependent on the input text.
Users can implement this feature by using the assistant_model parameter within the .generate() call in the Hugging Face Transformers library. In the Optimum Habana environment, this is triggered via the --assistant_model command-line parameter in text generation examples.
Hardware Context
Intel Gaudi processors are positioned as a cost-effective alternative for high-performance inference. According to Hugging Face, Intel Gaudi delivers performance similar to Nvidia H100 GPUs while maintaining a price point comparable to Nvidia A100 80GB GPUs.