GPT‑QModel – A unified platform for LLM quantisation

What it does

GPT‑QModel is a Python library that lets you compress large language models (LLMs) into smaller, faster‑running versions using a variety of quantisation techniques (GPTQ, AWQ, ParoQuant, GGUF, FP8, EXL3, QQQ, etc.). It provides a single, consistent API for:

• Calibration – measuring a model’s activation statistics on a sample dataset.
• Quantisation – converting weights to low‑bit formats (2‑8 bit, FP8, etc.).
• Validation – checking that the quantised model retains quality.
• Conversion & saving – exporting checkpoints in formats such as GGUF or MLX.
• Accelerated inference – loading the quantised model with popular runtimes like vLLM and SGLang.

All of this works across many hardware back‑ends: NVIDIA CUDA, AMD ROCm, Huawei Ascend, Intel XPU, Apple‑silicon Metal, and plain CPUs.

Key features (as described in the README)

Feature	Details
Broad method support	GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, GPTAQ, FOEM, BitsAndBytes, etc.
Multiple runtimes	Native integration with 🤗 Transformers, Optimum, PEFT, plus inference back‑ends vLLM and SGLang.
Hardware coverage	CUDA, ROCm, Ascend NPU, Intel XPU, Apple M‑series (Metal/MLX), and CPU (AVX/AMX).
Dynamic per‑layer quantisation	Each layer can have its own bit‑width or be excluded entirely.
Multi‑GPU data‑parallel quantisation	Up to ~80 % speed‑up when spreading work across GPUs.
Apple‑silicon native pipelines	MLX‑based GPTQ/AWQ quantisation and EXL3 weight‑quantisation on M‑series chips.
Extensive model catalogue	Supports >200 model families (Llama, Gemma, Qwen, DeepSeek, InternVL, etc.) via an internal MODEL_MAP.
Packaging API	hf_ packing/repacking API for layer‑wise kernel feasibility, used by tools like AutoRound.
High‑performance kernels	Marlin, Machete, BitBLAS, Triton, Exllama V3/EXL3, and Intel Torch‑fused kernels.
Quality‑focused	Unit‑test coverage, checkpoint validation, and tools for 4‑bit GPTQ evaluation.
Open‑source & PyPI	Apache‑2.0 licence, releases on GitHub and PyPI (pip install gptqmodel).

Who might use it?

• Research engineers wanting to experiment with low‑bit LLMs without stitching together many separate tools.
• MLOps teams that need a reproducible pipeline to quantise, validate, and deploy models on heterogeneous clusters.
• Product developers targeting edge devices (Apple Silicon, Intel CPUs, Ascend NPU) where memory is limited.
• Community contributors building new quantisation methods – the modular “method → format → backend” matrix makes it easy to plug in custom kernels.

How to get started (quick example from the README)

pip install 'gptqmodel[mlx]'   # optional MLX for Apple silicon

from gptqmodel import GPTQModel

# Quantise a Qwen model on Apple silicon using the native MLX pipeline
GPTQModel.quantize_mlx(
    "Qwen/Qwen3-0.6B",
    "./qwen3-mlx-gptq",
    method="gptq",   # or "awq"
)

The same API works for CUDA, ROCm, or CPU back‑ends; just install the appropriate extras and select the desired method and format.

Ecosystem & adoption

The README lists public references from Hugging Face, Intel, NVIDIA, and Alibaba Cloud, indicating that the library is being used in production‑grade pipelines and model‑hosting services.

---

All information above is taken directly from the repository’s README; no additional features have been inferred.