Q8-Chat: Efficient Generative AI on Intel Xeon CPUs

Hugging Face and Intel have demonstrated that utilizing SmoothQuant for 8-bit quantization allows large language models (LLMs) to run efficiently on Intel Xeon CPUs, reducing model size by approximately 2x and significantly lowering inference latency without compromising model quality.

Overcoming LLM Quantization Challenges with SmoothQuant

Standard quantization techniques often fail with LLMs because these models exhibit large-magnitude outliers in specific activation channels across all layers and tokens. When activations are quantized token-wise, these outliers typically lead to truncated values or underflowing low-magnitude activations, both of which significantly degrade model accuracy.

SmoothQuant addresses this issue by applying a joint mathematical transformation to weights and activations. This process reduces the ratio between outlier and non-outlier values for activations by increasing the ratio for weights, making the Transformer layers "quantization-friendly." This enables 8-bit integer (INT8) quantization, replacing complex 16-bit floating-point (FP16/BF16) operations with faster integer arithmetic.

Performance and Accuracy Benchmarks

Intel applied SmoothQuant-O3 to several LLMs, including OPT (2.7B and 6.7B), LLaMA (7B), Alpaca (7B), Vicuna (7B), BloomZ (7.1B), and MPT-7B-chat. The results indicate that 8-bit quantization maintains or improves model performance:

• OPT Models: These were identified as excellent candidates for SmoothQuant. Models were reduced in size by ~2x, with most benchmarks showing improvement and others seeing only marginal penalties.
• LLaMA 7B and BloomZ 7.1B: These models were compressed by ~2x, with approximately half of the tasks showing metric improvements. Only one task experienced a relative degradation greater than 3%.

Hardware Implementation and Q8-Chat

Efficient LLM deployment on CPUs is made possible by combining 8-bit quantization with 4th Gen Intel Xeon (Sapphire Rapids) hardware.

Q8-Chat is a demonstration hosted on Hugging Face Spaces that provides a ChatGPT-like experience running on a single-socket Intel Sapphire Rapids CPU with 32 cores and a batch size of 1. This setup demonstrates that high-quality chat experiences can be achieved on CPU platforms, providing organizations with greater IT flexibility and better cost-performance than relying solely on high-end GPUs or complex AI accelerators.

Integration with Hugging Face Ecosystem

To simplify the adoption of these techniques, Hugging Face is integrating these quantization methods into the Optimum Intel library via the Intel Neural Compressor. This integration aims to allow developers to replicate these efficiency gains with minimal code.