Accelerating StarCoder on Intel Xeon with Optimum Intel

Hugging Face and Intel have achieved over 7x inference acceleration for the StarCoder-15B model on 4th Generation Intel Xeon Scalable processors. This performance gain is realized by integrating 8-bit and 4-bit quantization techniques with assisted generation (speculative decoding) via the optimum-intel library.

Inference Performance Benchmarks

The integration of quantization and assisted generation significantly reduces Time Per Output Token (TPOT) while maintaining model accuracy on the HumanEval dataset.

StarCoder Quantization Precision HumanEval (pass@1) TTFT (ms) TTFT Speedup TPOT (ms) TPOT Speedup
Baseline None A16W16 33.54 357.9 1.00x 181.0 1.00x
INT8 SmoothQuant A8W8 33.96 163.4 2.19x 82.4 2.20x
INT4 RTN (g128) A16W4 32.80 425.1 0.84x 54.0 3.35x
INT8 + AG SmoothQuant A8W8 33.96 183.6 1.95x 24.8 7.30x

Quantization Strategies for Intel Xeon

To optimize StarCoder, the team addressed the memory bandwidth bottleneck—the primary constraint in autoregressive token generation—using two distinct quantization methods:

8-bit Static Quantization (INT8)

Using the SmoothQuant algorithm, the model is quantized to INT8 with minimal accuracy loss. SmoothQuant applies smoothing scaling factors to both activations and weights to mitigate the impact of large magnitude outliers in activations. This approach yielded a ~2.20x speedup in TPOT and a ~2.19x speedup in Time To First Token (TTFT).

4-bit Weight-Only Quantization (INT4)

To further reduce the memory footprint and loading time, the model weights were quantized to 4 bits using groupwise Round To Nearest (RTN) quantization (groups of 128). While this achieved a 3.35x speedup in TPOT, it introduced a 0.84x slowdown in TTFT due to the compute overhead required to dequantize 4-bit weights back to 16-bit before computation.

Assisted Generation (Speculative Decoding)

Assisted Generation (AG) accelerates inference by using a small, fast draft model to predict candidate tokens, which the larger target model then validates in parallel.

  • Draft Model: The team used bigcode/tiny_starcoder_py (164M parameters), which is approximately 95x smaller than the target StarCoder-15B model.
  • Optimal Configuration: The best results (~7.30x TPOT speedup) were achieved by applying 8-bit SmoothQuant to both the draft and target models.

Technical Trade-offs in AG

The choice of 8-bit quantization over 4-bit for Assisted Generation is driven by the shift in bottlenecks:

  1. Draft Model: An 8-bit quantized 164M parameter model fits largely within the CPU cache, eliminating the memory bandwidth bottleneck and avoiding the dequantization overhead associated with 4-bit WOQ.
  2. Target Model: Because the target model validates multiple tokens in parallel, the bottleneck shifts from memory bandwidth to computation. The 8-bit quantized model outperforms the 4-bit model because it avoids the compute-heavy dequantization process.

Implementation via Optimum Intel

Users can implement these optimizations by replacing the standard AutoModelForCausalLM class with IPEXModelForCausalLM from the optimum-intel library, which leverages the Intel Extension for PyTorch (IPEX) and Intel Advanced Matrix Extensions (AMX) on 4th Gen Xeon processors.

pip install --upgrade-strategy eager optimum[ipex]

from optimum.intel import IPEXModelForCausalLM
from transformers import AutoTokenizer, pipeline

model = IPEXModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

Sources