bitsandbytes-foundation/bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.

What it solves

It reduces the memory requirements for running and training large language models (LLMs), making them more accessible by allowing them to run on hardware with limited VRAM/RAM.

How it works

The library uses k-bit quantization to compress model weights and optimizer states. It provides three primary methods:

  • 8-bit Optimizers: Uses block-wise quantization to reduce the memory cost of optimizers while maintaining 32-bit performance.
  • LLM.int8(): An 8-bit quantization method for inference that uses vector-wise quantization for most features and 16-bit matrix multiplication for outliers to prevent performance loss.
  • QLoRA: A 4-bit quantization method for training that compresses the model to 4-bits and adds small, trainable low-rank adaptation (LoRA) weights.

Who it’s for

Developers and researchers who need to run or fine-tune large models on consumer-grade hardware or limited compute resources.

Highlights

  • Support for 4-bit and 8-bit quantization primitives.
  • Broad hardware support across Linux, Windows, and macOS, including NVIDIA, AMD, and Intel GPUs, as well as CPUs.
  • Integration with popular libraries like Hugging Face Transformers, Diffusers, and PEFT.

Related

  • Project
    pytorch/ao

    A PyTorch-native library for model optimization that provides quantization and sparsity tools to accelerate training and inference while reducing memory usage.

  • Project
    Xilinx/brevitas

    A PyTorch library for neural network quantization that supports both post-training quantization and quantization-aware training to improve model efficiency.

  • Project
    huggingface/optimum-quanto

    A PyTorch quantization backend for Optimum that reduces model size and memory usage by converting weights and activations to lower precision like int4 or float8.

  • Project
    intel/neural-compressor

    An open-source Python library that provides model compression and quantization techniques for PyTorch, TensorFlow, and JAX models to improve inference efficiency.

  • Project
    vllm-project/llm-compressor

    LLM Compressor is a Python library that quantizes and prunes large language models into the `compressed‑tensors` format, enabling memory‑efficient deployment with vLLM. It supports many low‑precision formats (NVFP4, FP8, INT4, etc.), several PTQ/GPTQ algorithms, DDP and disk‑offloading for huge models, and ships pre‑quantized checkpoints for popular LLMs.