FujitsuResearch/OneCompression

What it solves

OneCompression (OneComp) provides a fully automated pipeline for compressing Large Language Models (LLMs) to reduce their memory footprint and hardware requirements without significantly sacrificing accuracy. It simplifies the complex process of quantization—selecting bit-widths, applying algorithms, and evaluating results—into a streamlined workflow.

How it works

OneComp uses a variety of post-training quantization (PTQ) techniques to reduce the precision of model weights. It can automatically detect available GPU VRAM to determine the optimal bit-width per layer via its AutoBit feature. Key technical methods include:

• Quantization Error Propagation (QEP): Corrects errors by propagating them to subsequent layers.
• Layer-Projected Coordinate Descent (LPCD): A unified framework for optimizing quantization across arbitrary submodules.
• MDBF (Multi-Envelope Double Binary Factorization): Enables aggressive sub-1-bit compression using binary factorization.
• Rotation Preprocessing: Uses SpinQuant/OstQuant to learn optimal rotation matrices to reduce quantization error before the process begins.
• Post-Processing: Offers block-wise distillation and LoRA SFT fine-tuning to recover accuracy after quantization.

Who it’s for

It is designed for developers and researchers who want to deploy LLMs on local hardware with limited VRAM, as well as those looking to experiment with advanced quantization algorithms for high-efficiency model deployment.

Highlights

• One-line automation: Quantize models automatically using a simple CLI command or Python API.
• vLLM Integration: Built-in plugins allow quantized models to be served efficiently via vLLM.
• Mixed-Precision Support: Automatically assigns bit-widths per layer based on memory budgets.
• Broad Architecture Support: Verified for Llama, Qwen, Gemma, and GPT-OSS architectures.
• GGUF Export: Built-in support for converting models to GGUF format for use with llama.cpp and Ollama.