vllm-project/compressed-tensors

What it solves

compressed-tensors is a library designed to solve the fragmentation of model compression formats. Because different quantization and compression techniques (like GPTQ, AWQ, and SmoothQuant) often use their own unique storage formats and loading procedures, it is difficult for developers to switch between methods or support multiple techniques in a single inference engine.

How it works

The library extends the safetensors format to create a unified, extensible checkpoint format. It allows for the storage and management of compressed tensor data across a wide variety of schemes, including various bit-widths (INT8, FP8, MXFP4) and sparsity patterns. It provides tools to define quantization configurations, calculate scales, and compress models, which can be then saved and loaded seamlessly into inference engines like vLLM and SGLang.

Who it’s for

It is intended for AI researchers and developers who need to compress Large Language Models (LLMs) for efficient deployment, experiment with different quantization methods, and simplify their model deployment pipelines.

Highlights

• Unified Format: Supports multiple compression schemes (GPTQ, AWQ, SmoothQuant, etc.) in one consistent format.
• Flexible Quantization: Supports activation quantization, mixed precision, KV cache quantization, and non-uniform schemes where different layers are quantized differently.
• Sparsity and Transforms: Handles both unstructured and semi-structured (2:4) sparsity and rotation-based quantization transforms (e.g., Hadamard).
• Checkpoint Conversion: Ability to convert between various formats like AutoAWQ and ModelOpt NVFP4.
• Model Offloading: Includes transparent CPU, disk, and distributed offloading for models that exceed available VRAM.