Hugging Face integrates LLM.int8() 8-bit matrix multiplication into Transformers and Accelerate

TL;DR

Hugging Face announced that the 8‑bit LLM.int8() quantization method is now fully integrated into the transformers and accelerate libraries, allowing inference of massive models such as BLOOM‑176B with roughly half the memory footprint and no measurable loss in accuracy.

Why 8‑bit quantization matters for large language models

Large language models (LLMs) now exceed hundreds of billions of parameters (e.g., PaLM 540B, OPT 176B, BLOOM 176B). Storing a model in full‑precision FP32 requires 4 bytes per weight, leading to memory demands of several hundred gigabytes—far beyond the capacity of most GPUs. Reducing precision to half‑precision (FP16/BF16) halves the memory, but still leaves models like BLOOM 176B at ~350 GB. An additional 2× reduction is possible with 8‑bit integer (INT8) quantization, but naïve quantization traditionally degrades accuracy, especially for models larger than ~6 B parameters.

Core idea of LLM.int8(): zero‑degradation matrix multiplication

LLM.int8() overcomes the accuracy drop by treating outlier values separately:

  1. Outlier extraction – values whose magnitude exceeds a threshold (≈6) are identified per column of the hidden‑state matrix.
  2. Mixed‑precision matmul – outliers are multiplied in FP16, while the remaining bulk of the matrix is quantized to INT8 and multiplied using vector‑wise (row‑wise for activations, column‑wise for weights) quantization.
  3. Dequantization & aggregation – the INT8 result is dequantized back to FP16 and summed with the outlier FP16 result, yielding a final FP16 output.

This three‑step process preserves the exact inference quality of the original FP16/BF16 model while cutting memory usage to one‑quarter.

Quantization mechanics: zero‑point vs. absmax

  • Zero‑point quantization scales a floating‑point range (e.g., [-1, 1]) to the INT8 range [-127, 127] and rounds each value. The inverse scaling recovers an approximation of the original value.
  • Absmax quantization divides each tensor by its absolute maximum, multiplies by 127, and rounds. For a vector [1.2, ‑0.5, ‑4.3, …, 5.4] the scaling factor is 127/5.4 ≈ 23.5, producing integer values in [-127, 127].

Both schemes can be applied row‑wise or column‑wise, which is essential for accurate matrix multiplication at scale.

Empirical evidence of zero degradation

Benchmarks on OPT‑175B and BLOOM‑176B using the lm‑eval‑harness show that the absolute differences between INT8 and FP16/BF16 scores are below the standard error for all tasks (e.g., HellaSwag accuracy 0.7849 vs. 0.7849, Lambada perplexity 3.0142 vs. 3.0152). In one case (BLOOM‑176B on Lambada) the INT8 model performed slightly better. The paper LLM.int8(): 8‑bit Matrix Multiplication for Transformers at Scale provides the full evaluation.

Speed trade‑offs

Memory savings come with a modest slowdown for the largest models: BLOOM‑176B runs 15 %–23 % slower in INT8 than in FP16. Smaller models (e.g., T5‑3B, T5‑11B) initially suffered larger slowdowns, but recent optimizations reduced per‑token latency from 312 ms to 173 ms (T5‑3B) and from 45 ms to 25 ms (T5‑11B). Future releases aim to close the gap further.

Model Precision GPUs Tokens / ms (batch 1)
BLOOM‑176B BF16 8 × A100 80GB 239
BLOOM‑176B INT8 4 × A100 80GB 282
T5‑11B FP16 2 × T4 15GB 11.7
T5‑11B INT8 1 × T4 15GB 43.5

Integration into transformers

The key component is bitsandbytes.nn.Linear8bitLt, a drop‑in replacement for torch.nn.Linear. A minimal conversion workflow:

import torch, bitsandbytes as bnb
from bnb.nn import Linear8bitLt

# Define a FP16 model and save its weights
fp16 = torch.nn.Sequential(torch.nn.Linear(64, 64), torch.nn.Linear(64, 64))
torch.save(fp16.state_dict(), "model.pt")

# Build an INT8 version
int8 = torch.nn.Sequential(
    Linear8bitLt(64, 64, has_fp16_weights=False),
    Linear8bitLt(64, 64, has_fp16_weights=False),
)
int8.load_state_dict(torch.load("model.pt"))
int8 = int8.to(0)   # quantization occurs on GPU

After the .to call the weights are stored as int8 tensors in the range [-127, 127]. The original FP16 values can be recovered via (weight.CB * weight.SCB) / 127.

Leveraging accelerate for zero‑memory model construction

accelerate.init_empty_weights() creates a model on the meta device, allocating no RAM. The integration patches accelerate so that parameters retain their custom class (Int8Params) when moved off the meta device. A recursive helper replaces every nn.Linear with Linear8bitLt while preserving modules such as lm_head that should stay in full precision:

from accelerate import init_empty_weights
import torch.nn as nn, bitsandbytes as bnb

def replace_8bit_linear(model, threshold=6.0, exclude="lm_head"):
    for name, module in model.named_children():
        if list(module.children()):
            replace_8bit_linear(module, threshold, exclude)
        if isinstance(module, nn.Linear) and name != exclude:
            with init_empty_weights():
                model._modules[name] = bnb.nn.Linear8bitLt(
                    module.in_features,
                    module.out_features,
                    module.bias is not None,
                    has_fp16_weights=False,
                    threshold=threshold,
                )
    return model

Two PRs to accelerate ensure that set_module_tensor_to_device is called exactly once for each INT8 tensor, avoiding double‑quantization bugs.

Hardware and installation requirements

  • GPU support – INT8 tensor cores are required (NVIDIA Turing, Ampere, RTX 20/30, A40‑A100, T4). CPUs and older Kepler GPUs lack native support.
  • Installation – with Python ≥ 3.8:
pip install accelerate bitsandbytes
pip install git+https://github.com/huggingface/transformers.git

Demonstrations

Google Colab notebooks showcase running T5‑11B (originally 42 GB in FP32) in only 11 GB using INT8, and a BLOOM‑3B demo that fits comfortably on a single T4.

Future work and limitations

  • Speed for small models – ongoing work aims to bring INT8 latency on ≤6 B models on par with FP16.
  • Kepler GPU support – plans to add a separate software stack for GPUs lacking native INT8 tensor cores (e.g., GTX 1080).
  • State‑dict persistence – current INT8 checkpoints omit quantization statistics (CB, SCB), preventing direct loading from the Hub; adding this metadata is a priority.
  • CPU execution – no 8‑bit tensor core on CPUs; a future software path could broaden accessibility.
  • Beyond text – extending the technique to large vision, audio, and multimodal models is an open research direction.

Credits: Younes B., Tim Dettmers, and contributors listed in the original blog post.

Sources