bitsandbytes-foundation/bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.

What it solves

It reduces the memory requirements for running and training large language models (LLMs), making them more accessible by allowing them to run on hardware with limited VRAM/RAM.

How it works

The library uses k-bit quantization to compress model weights and optimizer states. It provides three primary methods:

  • 8-bit Optimizers: Uses block-wise quantization to reduce the memory cost of optimizers while maintaining 32-bit performance.
  • LLM.int8(): An 8-bit quantization method for inference that uses vector-wise quantization for most features and 16-bit matrix multiplication for outliers to prevent performance loss.
  • QLoRA: A 4-bit quantization method for training that compresses the model to 4-bits and adds small, trainable low-rank adaptation (LoRA) weights.

Who it’s for

Developers and researchers who need to run or fine-tune large models on consumer-grade hardware or limited compute resources.

Highlights

  • Support for 4-bit and 8-bit quantization primitives.
  • Broad hardware support across Linux, Windows, and macOS, including NVIDIA, AMD, and Intel GPUs, as well as CPUs.
  • Integration with popular libraries like Hugging Face Transformers, Diffusers, and PEFT.

Related

  • Project
    pytorch/ao

    TorchAO is a PyTorch-native quantization library that lets you compress and speed up models with minimal code changes. It supports float8 training, int4/int8 inference, and quantization-aware training to recover accuracy, delivering up to 1.5-2x speedups across popular workloads.

  • Project
    Xilinx/brevitas

    A PyTorch library for neural network quantization that supports both post-training quantization and quantization-aware training to optimize models for efficiency.

  • Project
    huggingface/optimum-quanto

    A PyTorch quantization backend for Optimum that reduces model size and memory usage by converting weights and activations to lower precision like int4 or float8.

  • Project
    intel/neural-compressor

    Intel Neural Compressor is an open‑source Python library for compressing PyTorch and JAX models. It supports static/dynamic quantization, FP8, weight‑only, SmoothQuant, mixed‑precision, and AutoRound‑driven tuning, targeting Intel CPUs, Xeon GPUs, and Habana Gaudi accelerators (with limited AMD/NVIDIA support). Install via `pip install neural-compressor-pt` (or `-jax`), prepare a model with a config, calibrate, and convert to a low‑precision version ready for fast inference.

  • Project
    vllm-project/llm-compressor

    LLM Compressor is a Python library that quantizes and compresses large language models for fast, low‑memory inference with vLLM. It supports many low‑bit formats (FP8, FP4, INT4), a variety of algorithms (GPTQ, AWQ, AutoRound, REAP), and can handle multi‑trillion‑parameter models via disk off‑loading and DDP. The library integrates directly with Hugging Face checkpoints, saves models in the `compressed‑tensors` format, and provides ready‑to‑use examples and pre‑compressed checkpoints.