vllm-project/compressed-tensors

A safetensors extension to efficiently store sparse quantized tensors on disk

What it solves

compressed-tensors is a library designed to solve the fragmentation of model compression formats. Different quantization and compression techniques (like GPTQ, AWQ, and SmoothQuant) often use proprietary or inconsistent storage formats, making it difficult for developers to and switch between them or deploy models across different inference engines.

How it works

It extends the safetensors format to create a unified, extensible checkpoint format. This allows it to represent a wide variety of compression schemes—including mixed precision, low-bit quantization, and sparsity patterns—within a single consistent structure. It also provides tools for checkpoint conversion and transparent offloading of models that exceed available VRAM to CPU or disk.

Who it’s for

AI researchers and developers who need to compress LLMs for efficient deployment, experiment with different quantization methods, and simplify their model deployment pipelines.

Highlights

  • Unified Format: Supports multiple compression schemes (GPTQ, AWQ, SmoothQuant, SparseGPT, etc.) in one consistent format.
  • Broad Quantization Support: Handles activation quantization, mixed precision, KV cache quantization, and non-uniform schemes where different layers are quantized differently.
  • Sparsity and Transforms: Supports both unstructured and semi-structured (2:4) sparsity and rotation-based quantization transforms (e.g., Hadamard).
  • Ecosystem Integration: Designed for seamless use with Hugging Face, PyTorch, vLLM, and SGLang.
  • Checkpoint Conversion: Ability to convert between various formats like AutoAWQ and ModelOpt NVFP4.

Related

  • Project
    ModelCloud/GPTQModel

    GPT‑QModel is an open‑source Python library that unifies many low‑bit quantisation methods (GPTQ, AWQ, etc.) for LLMs, offering calibration, conversion, validation, and fast inference across CUDA, ROCm, Ascend, Intel XPU, Apple Silicon and CPU, with integrations for Transformers, vLLM and SGLang.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Project
    vllm-project/llm-compressor

    A genuine, actively maintained Python library that quantizes and compresses large language‑model checkpoints (including MoE, vision‑language, and audio‑language models) for fast, low‑VRAM inference with vLLM. It offers many algorithms (GPTQ, AWQ, AutoRound, REAP), supports INT4/FP8/NVFP4/MXFP4 and mixed‑precision formats, provides Triton‑accelerated kernels, DDP and disk‑offloading for huge models, and outputs checkpoints in the `compressed‑tensors` format ready for vLLM.

  • Project
    intel/neural-compressor

    An open-source Python library that provides model compression and quantization techniques for PyTorch, TensorFlow, and JAX models to improve inference efficiency.

  • Project
    Tencent/AngelSlim

    A comprehensive toolkit for large model compression that provides unified access to quantization, speculative decoding, and sparse attention to reduce model size and increase inference speed.