z-lab/paroquant

[ICLR 2026] ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference

What it solves

ParoQuant addresses the accuracy loss typically associated with INT4 quantization in Large Language Models (LLMs), particularly for reasoning tasks. It aims to reduce the model size and memory footprint while maintaining performance close to the original full-precision (FP16) models.

How it works

The project implements Pairwise Rotation Quantization, which uses learned pairwise rotations to suppress weight outliers. This technique allows the model to be quantized to 4-bit (INT4) while closing the accuracy gap between the quantized version and the original FP16 model, all while maintaining high inference speeds similar to AWQ.

Who it’s for

It is designed for developers and researchers who want to deploy efficient, high-performance LLMs on various hardware, including NVIDIA GPUs (via vLLM and Transformers) and Apple Silicon (via MLX).

Highlights

  • High Accuracy: Closes the accuracy gap with FP16 for INT4 quantization.
  • Cross-Platform Support: Compatible with NVIDIA GPUs and Apple Silicon.
  • Broad Model Support: Provides pre-quantized checkpoints for Gemma 4, Qwen, and Llama families.
  • Flexible Deployment: Includes an OpenAI-compatible API server and interactive chat CLI.
  • Custom Quantization: Tools provided to optimize rotation parameters and quantize custom models.