Hugging Face AutoGPTQ and Transformers Integration
Hugging Face has integrated the AutoGPTQ library into the Transformers library, allowing users to quantize and run large language models (LLMs) in 8, 4, 3, or 2-bit precision. This integration significantly reduces the hardware requirements for deploying LLMs, enabling 4-bit quantization with negligible accuracy degradation and inference speeds comparable to the float16 (fp16) baseline for small batch sizes.
GPTQ Quantization Technical Overview
GPTQ is a Post-Training Quantization (PTQ) method that compresses pre-trained models using a calibration dataset and a few hours of computation. Unlike Quantization-Aware Training (QAT), which occurs during training, GPTQ allows for the compression of massive models without the need for expensive full-model retraining.
Mixed Precision Scheme
GPTQ utilizes a mixed int4/fp16 quantization scheme. In this architecture, weights are quantized as int4, while activations remain in float16. During inference, weights are dequantized on the fly within a fused kernel close to the compute unit, rather than in the GPU global memory. This provides two primary benefits:
- Memory Savings: Int4 quantization provides memory savings close to 4x.
- Inference Speed: Potential speedups are achieved by reducing the time spent on data communication due to lower bitwidth weights.
The Quantization Algorithm
GPTQ builds upon the Optimal Brain Quantization (OBQ) framework. It treats the compression of each layer as a problem of minimizing the mean squared error (MSE) between the original weight matrix and the quantized version. By employing per-channel quantization, GPTQ quantizes weights one at a time and updates remaining non-quantized weights using Hessian matrices to compensate for the error.
GPTQ optimizes this process to be significantly faster than OBQ. For example, a Bloom model (176B) can be quantized in less than 4 GPU-hours, whereas a BERT model (336M) took 2 GPU-hours using OBQ.
AutoGPTQ and Transformers Integration
AutoGPTQ is a library that provides broad coverage across various transformer architectures, making it a more versatile choice than architecture-specific implementations. Hugging Face has integrated a minimalist version of the AutoGPTQ API via the Optimum library to provide a native Transformers API for LLM quantization.
Native Support and Usage
Users can now run GPTQ models directly through AutoModelForCausalLM by installing AutoGPTQ and optimum. The integration supports both Nvidia GPUs and RoCm-powered AMD GPUs.
Key advantages of this integration include:
- Serializability: Quantized models can be saved and shared on the Hugging Face Hub.
- Performance: Inference latency is on par with FP16 inference for small batch sizes.
- Hardware Compatibility: Native support for AMD GPUs via RoCm.
- Kernel Support: Support for Exllama kernels across a wide range of architectures.
Performance Benchmarks
In benchmarks conducted on a single NVIDIA A100-SXM4-80GB GPU (prompt length 512, 512 generated tokens, batch size 1), the following results were observed:
| gptq | act_order | bits | group_size | kernel | Load time (s) | Per-token latency (ms) | Throughput (tokens/s) | Peak memory (MB) |
|---|---|---|---|---|---|---|---|---|
| False | None | None | None | None | 26.0 | 36.958 | 27.058 | 29152.98 |
| True | False | 4 | 128 | exllama | 36.2 | 33.711 | 29.663 | 10484.34 |
| True | False | 4 | 128 | autogptq-cuda-old | 36.2 | 46.44 | 21.53 | 10344.62 |
Deployment and Fine-Tuning
Text-Generation-Inference (TGI)
GPTQ support has been added to the Text-Generation-Inference (TGI) library for production serving. This allows for the deployment of very large models on limited hardware; for instance, a 70B model can now be served on a single A100-80GB GPU, which would be impossible with an fp16 checkpoint.
Fine-Tuning with PEFT
While quantized models cannot be trained using standard methods, users can leverage the PEFT (Parameter-Efficient Fine-Tuning) library to train adapters on top of a frozen quantized model.
Current Limitations and Future Directions
Supported Architectures
Currently, the integration supports large language models with decoder-only or encoder-only architectures (e.g., Llama, OPT, GPT-Neo, GPT-NeoX). Vision, audio, and multi-modal models are not yet supported.
Areas for Improvement
- Kernel Optimization: While Exllama kernels are used, there is potential for further improvement through Triton-based kernels or other implementations from MIT Han Lab or Kim et al.
- Weight and Activation Quantization: GPTQ only quantizes weights. Future directions include exploring W4A8 quantization kernels that could utilize integer arithmetic via Nvidia Tensor Cores.
- Batch Size Scaling: Designing performant W4A16 kernels that scale efficiently with larger batch sizes remains an open challenge.