bitsandbytes-foundation/bitsandbytes
Accessible large language models via k-bit quantization for PyTorch.
What it solves
It reduces the memory requirements for running and training large language models (LLMs), making them more accessible by allowing them to run on hardware with limited VRAM/RAM.
How it works
The library provides k-bit quantization primitives for PyTorch, using three primary methods:
- 8-bit Optimizers: Uses block-wise quantization to maintain performance while significantly reducing the memory cost of optimizers.
- LLM.int8(): Enables 8-bit quantization for inference, using vector-wise quantization for most features and 16-bit matrix multiplication for outliers to prevent performance loss.
- QLoRA: Implements 4-bit quantization for training, combining quantization with low-rank adaptation (LoRA) weights to allow efficient fine-tuning.
Who it’s for
Developers and researchers working with PyTorch who need to deploy or fine-tune large models on consumer-grade or limited-resource hardware.
Highlights
- Support for 4-bit and 8-bit quantization primitives.
- Broad hardware accelerator support including NVIDIA, AMD, Intel GPUs, and CPUs across Linux, Windows, and macOS.
- Integration with the Hugging Face ecosystem (Transformers, Diffusers, PEFT).
- Memory-saving techniques that maintain model performance.