ModelCloud/GPTQModel

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

What it solves

GPT-QModel is a toolkit for LLM model quantization (compression), allowing large models to run on hardware with limited memory by reducing the precision of model weights. It addresses the challenge of making high-performance models accessible across a wide variety of hardware accelerators and CPUs.

How it works

The toolkit provides a suite of quantization methods (including GPTQ, AWQ, ParoQuant, GGUF, FP8, and EXL3) and hardware-optimized kernels. It supports a vast range of hardware, including NVIDIA CUDA, AMD ROCm, Huawei Ascend NPU, Intel XPU, and various CPUs (Intel/AMD/Apple). It integrates with popular frameworks like Hugging Face Transformers, vLLM, and SGLang to ensure that quantized models can be easily loaded and executed.

Who it’s for

It is designed for AI developers and researchers who need to compress large language models for deployment on diverse hardware, as well as those looking to optimize inference speed and VRAM usage for MoE (Mixture of Experts) and multimodal models.

Highlights

  • Broad Hardware Support: Native acceleration for NVIDIA, AMD, Huawei Ascend, Intel XPU, and Apple Silicon (via MLX).
  • Diverse Quantization Methods: Supports GPTQ, AWQ, ParoQuant, GGUF, FP8, EXL3, and QQQ.
  • MoE Optimization: Features data-parallel quantization for MoE models on multi-GPU setups and specialized routing controls to improve quantization quality.
  • Multimodal Support: Optimized quantization for image-to-text models, utilizing image calibration data for better results.
  • Framework Integration: Seamless compatibility with Hugging Face, vLLM, SGLang, PEFT, and Optimum.

Related

  • Project
    vllm-project/llm-compressor

    A library for quantizing and compressing large language models (weights, activations, KV‑cache, etc.) so they run efficiently with the vLLM inference engine. Supports many algorithms (GPTQ, AWQ, AutoRound, REAP pruning, etc.) and formats (FP8, NVFP4, INT4, mixed‑precision). Provides one‑line API, DDP + disk‑offloading for multi‑terabyte models, and pre‑quantized checkpoints for popular LLMs.

  • Project
    FujitsuResearch/OneCompression

    A Python package for automated LLM compression that uses advanced quantization techniques like QEP and AutoBit to reduce model size and VRAM usage.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.

  • Project
    Picovoice/picollm

    A cross-platform inference engine for running highly accurate, compressed LLMs locally on devices ranging from mobile phones to web browsers.

  • Dispatch
    Hugging Face AutoGPTQ and Transformers Integration

    Hugging Face has integrated the AutoGPTQ library into Transformers, enabling the quantization of LLMs to 8, 4, 3, or 2-bit precision to reduce memory requirements with negligible accuracy loss at 4-bit.