FujitsuResearch/OneCompression

Python package for LLM compression

Fujitsu One Compression (OneComp) – What It Is

OneComp is a Python library that lets you shrink large language models (LLMs) so they fit into the memory of your GPU (or even a CPU/MPS device) without losing much accuracy. You run it with a single command like:

onecomp meta-llama/Llama-2-7b-hf

or from Python via Runner.auto_run(). The tool automatically:

  1. Detects how much VRAM you have.
  2. Chooses a per‑layer bit‑width (e.g., 4‑bit, 3‑bit, sub‑1‑bit) that fits.
  3. Quantizes the model using sophisticated post‑training techniques.
  4. Evaluates the quantized model and writes the result to disk.

Core Features (as described in the README)

Feature What It Does
Quantization Error Propagation (QEP) After a standard post‑training quantization, QEP pushes the quantization error forward through the network so later layers can compensate, improving accuracy.
Layer‑Projected Coordinate Descent (LPCD) A flexible PTQ framework that can quantize arbitrary sub‑modules by solving a relaxed optimization problem and projecting the solution back onto per‑layer quantizers.
AutoBit Mixed‑precision quantization that formulates bit‑width assignment as an integer‑linear program, automatically picking the best per‑layer bits to meet a VRAM budget.
JointQ Simultaneously optimises weight quantization and scaling factors, supporting group‑wise quantization (e.g., 4‑bit with groupsize = 128).
MDBF (Multi‑Envelope Double Binary Factorization) An aggressive binary‑factor quantizer that represents each weight matrix as a sum of signed binary matrices plus FP16 “envelopes”, enabling sub‑1‑bit compression with optional ADMM/gradient refinement.
Block‑wise PTQ Distills a quantized model block‑by‑block against an FP16 teacher, reducing intermediate‑representation error.
LoRA SFT Post‑Process Fine‑tunes a quantized model with LoRA adapters to recover accuracy or add domain knowledge.
Rotation Preprocessing Learns a rotation matrix before quantization (SpinQuant/OstQuant) to reduce error; the rotation is baked into the weights at load time.
vLLM Plugin Integration Built‑in plugins let you serve quantized models with the high‑performance vLLM inference engine, and you can pair this with Open WebUI for a local ChatGPT‑style UI.
GGUF Export & Hugging Face Hub Converts models to the GGUF v3 format (used by llama.cpp/Ollama) and can push the quantized checkpoint to the Hugging Face Hub, complete with a model card describing the quantization recipe.
Web Dashboard (HPC) A browser‑based UI for launching quantization jobs on HPC clusters, monitoring progress, and testing chat inference.

Supported Model Families

The library has been tested on Llama (1‑3), Qwen3 (up to 32 B), Gemma (2‑4), GPT‑OSS (20 B & 120 B), and Qwen3.6 (up to 35 B). Other Hugging Face‑compatible models may work but are not officially verified.

Installation & Usage

  • For end‑users – install PyTorch (CPU, CUDA, or macOS MPS) then pip install onecomp. Add the optional distributed extra for multi‑GPU DeepSpeed support.
  • For developers – the repo recommends the Rust‑based uv tool to create a reproducible environment, with extras for CUDA versions, vLLM, visualization, docs, etc.
  • Running – the CLI (onecomp <model-id>) or the Python API (Runner.auto_run) handles everything from calibration data loading to saving the quantized checkpoint.

Documentation & Resources

  • Full docs: https://FujitsuResearch.github.io/OneCompression/
  • Tutorial notebook (Jupyter/Colab) walks through RTN visualisation, Runner.auto_run, and vLLM chat inference.
  • Example scripts cover every feature (GPTQ, QEP, LPCD, AutoBit, LoRA SFT, vLLM serving, etc.).
  • Academic references: NeurIPS 2025 paper on QEP and an arXiv 2025 paper on LPCD.

Who Might Use It?

  • Researchers needing to evaluate low‑bit quantization methods on LLMs.
  • Engineers who want to run large models on a single GPU or on CPU/MPS machines.
  • Anyone looking to deploy a quantized model locally with vLLM and a web UI.

Bottom line: OneComp is a genuine, production‑grade toolkit for post‑training quantization of large language models, offering a range of advanced algorithms and seamless integration with popular inference stacks.

Related

  • Project
    vllm-project/llm-compressor

    A genuine, actively maintained Python library that quantizes and compresses large language‑model checkpoints (including MoE, vision‑language, and audio‑language models) for fast, low‑VRAM inference with vLLM. It offers many algorithms (GPTQ, AWQ, AutoRound, REAP), supports INT4/FP8/NVFP4/MXFP4 and mixed‑precision formats, provides Triton‑accelerated kernels, DDP and disk‑offloading for huge models, and outputs checkpoints in the `compressed‑tensors` format ready for vLLM.

  • Project
    Tencent/AngelSlim

    A comprehensive toolkit for large model compression that provides unified access to quantization, speculative decoding, and sparse attention to reduce model size and increase inference speed.

  • Project
    Picovoice/picollm

    A cross-platform inference engine for running highly accurate, compressed LLMs locally on devices ranging from mobile phones to web browsers.

  • Project
    InternLM/lmdeploy

    A toolkit for compressing, deploying, and serving LLMs and VLMs, offering high-performance inference engines and quantization to maximize throughput.

  • Project
    localai-org/apex-quant

    A MoE-aware mixed-precision quantization technique for llama.cpp that reduces MoE model size and increases speed while maintaining high accuracy.