z-lab/paroquant

[ICLR 2026] ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference

What it solves

ParoQuant addresses the accuracy loss typically associated with INT4 quantization in Large Language Models (LLMs), particularly for reasoning tasks. It aims to close the performance gap between low-precision (INT4) and high-precision (FP16) models while maintaining high inference speeds.

How it works

The project implements Pairwise Rotation Quantization, which uses learned pairwise rotations to suppress weight outliers. This technique allows the model to be compressed to 4-bit precision without the significant accuracy drops often seen in standard quantization methods.

Who it’s for

It is designed for developers and researchers who need to run large models efficiently on limited hardware, specifically those using NVIDIA GPUs (via vLLM and Transformers) or Apple Silicon (via MLX).

Highlights

  • High Efficiency: Runs at speeds comparable to AWQ.
  • Broad Hardware Support: Compatible with NVIDIA GPUs and Apple Silicon.
  • Pre-quantized Models: Provides a wide collection of ParoQuant-optimized versions of Qwen, Gemma, and Llama models on Hugging Face.
  • Custom Quantization: Includes tools to optimize rotation parameters and convert existing models into ParoQuant checkpoints.

Related

  • Project
    ModelCloud/GPTQModel

    A comprehensive LLM quantization toolkit that compresses models using methods like GPTQ and AWQ with hardware acceleration for NVIDIA, AMD, Intel, and Huawei NPUs.

  • Dispatch
    Intel AutoRound: Advanced Weight-Only Quantization for LLMs and VLMs

    Intel has introduced AutoRound, a weight-only post-training quantization method that uses signed gradient descent to enable high-accuracy low-bit quantization (INT2-INT8) for LLMs and VLMs.

  • Project
    intel/auto-round

    AutoRound is Intel’s open‑source Python toolkit for ultra‑low‑bit (2‑4 bit) quantization of large language and vision‑language models. It offers high‑accuracy rounding (sign‑gradient descent), multiple export formats (AutoRound, AutoAWQ, AutoGPTQ, GGUF), and seamless integration with Transformers, vLLM, SGLang, and LLM‑Compressor. Install via `pip install auto-round` (or the HPU/XPU variants) and quantize models through a simple CLI (`auto-round …`) or Python API (`AutoRound(...).quantize_and_save`).

  • Project
    Picovoice/picollm

    A cross-platform inference engine for running highly accurate, compressed LLMs locally on devices ranging from mobile phones to web browsers.

  • Project
    FujitsuResearch/OneCompression

    A Python package for automated LLM compression that uses advanced quantization techniques like QEP and AutoBit to reduce model size and VRAM usage.