bitsandbytes: a k-bit quantization library for PyTorch that reduces memory consumption for LLM inference and training

What it solves

It reduces the memory requirements for running and training large language models (LLMs), making them more accessible by allowing them to run on hardware with limited VRAM/RAM.

How it works

The library provides k-bit quantization primitives for PyTorch, using three primary methods:

• 8-bit Optimizers: Uses block-wise quantization to maintain performance while significantly reducing the memory cost of optimizers.
• LLM.int8(): Enables 8-bit quantization for inference, using vector-wise quantization for most features and 16-bit matrix multiplication for outliers to prevent performance loss.
• QLoRA: Implements 4-bit quantization for training, combining quantization with low-rank adaptation (LoRA) weights to allow efficient fine-tuning.

Who it’s for

Developers and researchers working with PyTorch who need to deploy or fine-tune large models on consumer-grade or limited-resource hardware.

Highlights

• Support for 4-bit and 8-bit quantization primitives.
• Broad hardware accelerator support including NVIDIA, AMD, Intel GPUs, and CPUs across Linux, Windows, and macOS.
• Integration with the Hugging Face ecosystem (Transformers, Diffusers, PEFT).
• Memory-saving techniques that maintain model performance.