vllm-project/llm-compressor

📦 llmcompressor – Fast, flexible quantization for LLMs on vLLM

What it is – llmcompressor is a Python library that prepares large language‑model checkpoints for efficient inference with the vLLM runtime. It does this by applying a wide range of quantization algorithms (weight‑only, activation, KV‑cache, attention) and model‑level transforms, then saving the result in the compressed‑tensors format that vLLM can load directly.

Why it matters – Quantization can shrink a 30‑B‑parameter model to a fraction of its original VRAM footprint while keeping most of the original quality. llmcompressor adds:

Speedy kernels – a Triton‑based GPTQ kernel that is ~15× faster than the previous eager implementation and can batch layers of the same shape.
Many precision formats – INT4, FP8, NVFP4, MXFP4, mixed‑precision combos (e.g., W4A16, NVFP4A16), and low‑bit custom formats.
Support for exotic architectures – MoE models, vision‑language, audio‑language, and even model‑free definitions.
Scalable workflows – DDP and disk‑offloading let you compress multi‑hundred‑billion‑parameter models on a single GPU or a small GPU cluster.

🎯 Typical use‑cases

Situation	How llmcompressor helps
Deploy a 30‑B‑parameter LLM on a single GPU	Convert weights/activations to FP8 or NVFP4, reducing VRAM > 70 % while preserving perplexity.
Run a massive MoE model (e.g., Qwen‑3.8‑2.4T) on limited hardware	Use the REAP expert‑pruning + NVFP4 pipeline to drop the least‑salient experts before quantization.
Need a fast quantization pipeline for many models	Triton‑accelerated GPTQ and batched processing cut end‑to‑end time from days to hours.
Want to keep the original Hugging‑Face workflow	The library works directly with `AutoModelForCausalLM` / `AutoTokenizer` and outputs a standard `compressed‑tensors` checkpoint that vLLM loads with `LLM(<path>)`.

⚙️ Core features (as listed in the README)

Quantization algorithms: Simple PTQ, GPTQ, AWQ, SmoothQuant, AutoRound, SpinQuant, QuIP, REAP expert pruning.
Supported precisions: W8A8, W4AFP8, NVFP4, MXFP4, MXFP8, FP8, INT4, plus arbitrary‑bit (WNA4/8/16).
Mixed‑precision pipelines: Combine weight‑only (e.g., INT4) with activation‑only (FP8) or per‑block formats.
KV‑cache & attention quantization: FP8 and NVFP4 options, including per‑head variants.
Architecture‑specific support: MoE LLMs, vision‑language, audio‑language, and models without a Hugging‑Face definition.
Scalable execution: Distributed Data Parallel (DDP) quantization, sequential on‑loading, and disk‑offloading for models that don’t fit in GPU memory.
Seamless vLLM integration: Produced checkpoints are directly consumable by vllm.LLM.

📦 Installation

pip install llmcompressor   # pulls the library and its dependencies

(For inference with the resulting checkpoint, install vllm separately.)

🛠️ Quick‑start example (from the README)

from compressed_tensors.offload import dispatch_model
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "Qwen/Qwen3-30B-A3B"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Quantize weights & activations to FP8 (block‑wise) and skip a couple of layers
recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_BLOCK",
    ignore=["lm_head", "re:.*mlp.gate$"],
)

oneshot(model=model, recipe=recipe)   # applies the quantization in‑place

# Verify a short generation
dispatch_model(model)
ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to(model.device)
print(tokenizer.decode(model.generate(ids, max_new_tokens=20)[0]))

# Save for vLLM
SAVE_DIR = "Qwen3-30B-A3B-FP8-BLOCK"
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)

Then run with vLLM:

from vllm import LLM
llm = LLM("Qwen3-30B-A3B-FP8-BLOCK")
print(llm.generate("My name is"))

📚 Documentation & community

Full docs – https://docs.vllm.ai/projects/llm-compressor/en/latest/ (installation, step‑by‑step compression guide, FAQs).
Blog announcement – https://neuralmagic.com/blog/llm-compressor-is-here-faster-inference-with-vllm/.
Community – vLLM Slack (#sig-quantization, #llm-compressor).
Survey – a 1‑minute user survey linked in the README for feedback.

📈 Project maturity

Published on PyPI (llmcompressor), version ≥ 0.13.0.
Actively maintained by Red Hat AI and the vLLM Project.
Provides many production‑ready checkpoints (GLM‑5.3, Qwen 3.8, Nemotron 3.5, Muse‑Glimmer, Kimi‑K3, etc.) hosted on Hugging Face.
Includes extensive example scripts covering every quantization path.

🧩 Ecosystem fit

llmcompressor sits between model preparation and runtime inference:

Input – a Hugging Face checkpoint (any architecture supported).
Processing – quantization/pruning via the library’s APIs.
Output – a compressed‑tensors checkpoint.
Runtime – load the checkpoint with vLLM for ultra‑fast serving.

📜 Citation

If you use the library in research or a product, cite:

@software{llmcompressor2024,
    title={{LLM Compressor}},
    author={Red Hat AI and vLLM Project},
    year={2024},
    month={8},
    url={https://github.com/vllm-project/llm-compressor},
}

Bottom line – llmcompressor is a production‑grade, highly optimized toolkit for turning large, often unwieldy LLM checkpoints into compact, vLLM‑ready artifacts, supporting the latest quantization research (GPTQ, AWQ, AutoRound, REAP, etc.) and scaling to multi‑hundred‑billion‑parameter models.