Hugging Face Transformers Quantization Overview

Hugging Face natively supports two primary quantization schemes in the Transformers library: bitsandbytes and auto-gptq. These integrations allow users to run large models on smaller hardware and perform parameter-efficient fine-tuning (PEFT) using adapters.

Comparing bitsandbytes and auto-gptq

Choosing between bitsandbytes and auto-gptq depends on whether the primary goal is ease of setup and fine-tuning or maximum inference speed for text generation.

bitsandbytes: Ease of Use and Fine-tuning

Bitsandbytes is designed for accessibility and flexibility, offering several key advantages:

  • Zero-shot Quantization: It does not require a calibration dataset, meaning any model containing torch.nn.Linear modules can be quantized out of the box during model load.
  • Cross-modality Interoperability: Because it targets linear layers, it works across different modalities, supporting models like Whisper, ViT, and Blip2.
  • Adapter Integration: Adapters trained on a quantized base model can be merged back into the base model or a dequantized model for deployment without inference performance degradation.

Limitations: bitsandbytes 4-bit models are slower than GPTQ during text generation and currently do not support 4-bit weight serialization.

auto-gptq: Optimized Inference

Auto-gptq is optimized for deployment and high-throughput text generation:

  • Generation Speed: GPTQ quantized models are significantly faster for text generation than bitsandbytes models.
  • N-bit Support: The algorithm supports quantization down to 2 bits, though 4 bits is the recommended tradeoff for quality.
  • Serialization: GPTQ models support serialization for any number of bits, allowing for easy loading of pre-quantized models (such as those from TheBloke).
  • Hardware Support: Integration works out of the box for AMD GPUs.

Limitations: GPTQ requires a calibration dataset for quantization, which can take several hours (e.g., 4 GPU hours for a 175B parameter model). Additionally, the current API is designed specifically for language models and does not natively support multimodal models.

Performance and Speed Benchmarks

Hugging Face conducted benchmarks using meta-llama/Llama-2-7b-hf and meta-llama/Llama-2-13b-hf on NVIDIA A100, T4, and Titan RTX GPUs.

Inference and Generation Speed

  • Forward Pass (Prefill): For the prefill step, bitsandbytes and GPTQ perform similarly, though GPTQ is slightly faster at larger batch sizes.
  • Text Generation: GPTQ is consistently faster than bitsandbytes across different hardware (A100, T4, Titan RTX) and generation lengths. When using attention caching (use_cache=True) with a batch size of 4, GPTQ can be twice as fast as bitsandbytes.

Adapter Fine-tuning Speed

When fine-tuning adapters using Low Rank Adapters (LoRA), bitsandbytes is faster than GPTQ. Note that for GPTQ fine-tuning, exllama kernels must be disabled as they are not supported for training.

Model Quality Degradation

Benchmarks from the Open-LLM leaderboard indicate that quantization causes minimal performance degradation, and this degradation is less pronounced in larger models.

Llama-2-7b Performance (Average Score):

  • FP16: 54.32
  • bnb-4bit: 53.4
  • GPTQ: 53.23

Llama-2-13b Performance (Average Score):

  • FP16: 58.66
  • GPTQ (actorder_True): 58.03
  • GPTQ: 57.56
  • bnb-4bit: 56.9

Recommended Workflow for Optimized Models

To achieve the best balance of fine-tuning flexibility and deployment performance, Hugging Face suggests the following pipeline:

  1. Quantize the base model using bitsandbytes for zero-shot quantization.
  2. Fine-tune adapters on top of the quantized base model.
  3. Merge the trained adapters into the base model or a dequantized model.
  4. Quantize the resulting merged model using GPTQ for final deployment to maximize generation speed.

Sources