vllm-project/llm-compressor

📦 llmcompressor – Fast LLM Quantization for vLLM

What it is – llmcompressor is a Python library that lets you compress (quantize) large language models so they run faster and use far less GPU memory when served with the vLLM inference engine. It supports a wide range of quantization schemes (int8, fp8, NVFP4, INT4, mixed‑precision, etc.), works directly with Hugging Face checkpoints, and can handle extremely large models via distributed data‑parallel (DDP) training and disk‑offloading.

---

🎯 Core capabilities

Capability	Details
Quantization algorithms	PTQ, GPTQ, AWQ, SmoothQuant, AutoRound, SpinQuant, QuIP, REAP expert pruning
Supported precisions	Activation: W8A8, W4AFP8, NVFP4, MXFP4/8, etc.; Mixed‑precision: W4A16, W8A16, MXFP8A16, NVFP4A16; Low‑bit: INT4, WNA4/8/16
What can be quantized	Weights, activations, KV‑cache, attention tensors; also MoE layers, vision‑language and audio‑language models
Model formats	Saves to the compressed‑tensors format, which vLLM can load directly
Scalability	DDP + disk offloading lets you compress multi‑terabyte models on a single GPU or a small GPU cluster
Integration	One‑line API (oneshot) works with any AutoModelForCausalLM from 🤗 Transformers

---

🚀 Getting started (quick example)

from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from compressed_tensors.offload import dispatch_model

MODEL_ID = "Qwen/Qwen3-30B-A3B"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID)
 tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_BLOCK",
    ignore=["lm_head", "re:.*mlp.gate$"],
)

oneshot(model=model, recipe=recipe)          # apply FP8 quantization

# sanity‑check generation
dispatch_model(model)
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to(model.device)
print(tokenizer.decode(model.generate(input_ids, max_new_tokens=20)[0]))

# save for vLLM
SAVE_DIR = "Qwen3-30B-FP8-BLOCK"
model.save_pretrained(SAVE_DIR)
 tokenizer.save_pretrained(SAVE_DIR)

Load the compressed checkpoint with vLLM:

pip install vllm

from vllm import LLM
llm = LLM("Qwen3-30B-FP8-BLOCK")
print(llm.generate("My name is"))

---

📚 Documentation & resources

• Full docs – https://docs.vllm.ai/projects/llm-compressor/en/latest/
• Installation – pip install llmcompressor
• Step‑by‑step guide – Choosing model, algorithm, and precision: https://docs.vllm.ai/projects/llm-compressor/en/latest/steps/choosing-model/
• Examples – Over 30 ready‑to‑run scripts covering weight‑only, activation, KV‑cache, MoE, multimodal, non‑uniform, and distributed quantization (see the examples/ folder).
• Blog announcement – https://neuralmagic.com/blog/llm-compressor-is-here-faster-inference-with-vllm/

---

🤝 Community & contribution

• Slack – Join the vLLM community (#sig-quantization, #llm-compressor).
• Issues & PRs – Open on GitHub: https://github.com/vllm-project/llm-compressor/issues
• Contributing guide – CONTRIBUTING.md in the repo.
• User survey – 1‑minute feedback form linked in the README.

---

📦 Release info

• PyPI – llmcompressor (latest version shown on badge).
• Pre‑quantized checkpoints – Red Hat AI provides FP8/NVFP4/INT4 checkpoints for models such as Qwen‑3.8, Nemotron 3.5 Lightning, Muse‑Glimmer‑30B, Kimi‑K3, Hy3, GLM‑5.2, etc.

---

📖 Citation

If you use the library in research, cite:

@software{llmcompressor2024,
    title={{LLM Compressor}},
    author={Red Hat AI and vLLM Project},
    year={2024},
    month={8},
    url={https://github.com/vllm-project/llm-compressor},
}

---

TL;DR

llmcompressor is a production‑grade, open‑source toolbox that turns massive LLM checkpoints into compact, fast‑inference models compatible with vLLM, supporting many quantization algorithms, mixed‑precision formats, and large‑scale distributed workflows.