Quantization-Aware Healing enables a 4-bit LLM that outperforms its full‑precision original

TL;DR

Quantization‑Aware Healing (QAH) compresses a 120B‑parameter GPT‑OSS model to 60B parameters and 4‑bit MXFP4 precision, and the resulting model outperforms its full‑precision bfloat16 counterpart on 7 of 9 benchmarks, proving that 4‑bit models can be both cheaper and more accurate.

Why conventional healing falls short after structural compression

• Conventional pipelines compress the architecture, quantize the weights, then apply a healing step.
• Quantization‑Aware Training (QAT) adds fake‑quantization operators and fine‑tunes on a task loss. It is expensive and can become unstable if training continues past the optimal point.
• Quantization‑Aware Distillation (QAD) distills a full‑precision teacher into the quantized student using KL‑divergence. This works only when the teacher shares the same architecture; after structural compression, no full‑precision version of the smaller architecture exists, so the teacher is a degraded recovered checkpoint, capping accuracy.
• Consequently, the field lacked a method to heal models that have been both structurally compressed and quantized.

The Quantization‑Aware Healing (QAH) recipe

• QAH distills directly from the original, pre‑compression model (the 120B teacher) into the compressed 4‑bit student, even though they have different architectures.
• The student receives only the teacher’s output distribution, matched via KL‑divergence on logits—no hard labels are used.
• This treats quantization as a second distillation pass rather than a lossy post‑processing step, allowing the student to acquire information missed during earlier recovery.
• KL‑based distillation provides stability: once the student matches the frozen teacher distribution, there is no pressure to drift, unlike cross‑entropy task loss which can cause degradation.
• For long‑context healing (up to 32k tokens), QAH reuses a memory‑efficient chunked KL‑divergence loss that processes one slice at a time, fitting within a fixed GPU memory budget.

Figure 1: QAH restores performance by distilling from the original model rather than the recovered checkpoint.

Empirical results on a 60B‑parameter 4‑bit model

The QAH pipeline was applied to a GPT‑OSS 120B model compressed to 60B parameters, recovered in bfloat16, then re‑quantized to MXFP4. Performance was compared against the same 60B bfloat16 checkpoint and the original 120B teacher.

Benchmark	120B teacher (MXFP4)	60B BF16 (recovered)	60B MXFP4 (QAH)	QAH vs BF16
AA‑LCR (long‑context reasoning)	50.0	35.3	42.7	+7.4
AIME 2025 (math)	80.0	70.7	76.3	+5.6
Aider (agentic coding)	45.3	38.2	40.9	+2.7
τ²‑bench (tool use)	68.4	59.4	61.7	+2.3
GPQA Diamond (science)	69.0	65.7	67.4	+1.7
IFBench (instruction following)	63.3	58.4	59.9	+1.5
LiveCodeBench (coding)	66.0	65.5	66.5	+1.0
MMLU‑Pro (knowledge)	78.0	74.0	73.8	–0.2
SciCode (science coding)	37.5	35.6	34.2	–1.4

• The 4‑bit QAH model beats its own bfloat16 source on 7 of 9 benchmarks.
• Gains are largest on tasks most harmed by compression: long‑context reasoning (+7.4) and mathematics (+5.6).
• Against the original 120B teacher, the QAH model matches or exceeds performance on LiveCodeBench and stays within 1.6 points on GPQA Diamond, despite using half the parameters and roughly a quarter of the weight memory.

Figure 2: The QAH model matches or beats its bfloat16 source on most benchmarks and competes closely with the full‑size teacher.

QAH vs. QAT: speed and stability comparison

• Experiment: quantize a GPT‑OSS 9B model to MXFP4 and train with either QAH or QAT.
• Peak accuracy: QAH reaches 54.9, QAT reaches 54.6 – essentially tied.
• Training efficiency: QAH peaks after ~100 steps (≈7× faster than QAT’s ~700 steps).
• Stability: QAH remains within ~2 points of its peak through 1,200 steps; QAT collapses, losing ~19 points after its peak.
• Practical impact: QAT requires careful early‑stopping to avoid degradation, whereas a fully trained QAH checkpoint can be deployed safely without drift.

Figure 3: QAH achieves rapid convergence and maintains stability, unlike QAT which degrades after its peak.

Practical implications for deployment

• Memory efficiency: 4‑bit precision reduces weight memory by ~4× compared to bfloat16.
• Compute reduction: Halving the parameter count roughly halves per‑token compute; combined with 4‑bit precision, the effective compute cost can be up to 8× lower than a bfloat16 baseline.
• Cost‑performance trade‑off: A QAH‑trained 4‑bit model delivers higher accuracy while requiring smaller hardware, turning quantization from a performance tax into an additional learning signal.
• Workflow simplification: QAH eliminates the need to re‑run the full multi‑stage post‑training pipeline (supervised fine‑tuning, RLHF, agentic tuning) after quantization, saving both time and compute.

Future directions

• The method is part of Multiverse Computing’s broader effort to make large models cheaper without sacrificing capability, complementing their work on efficient knowledge distillation for long‑context training.
• Open questions include scaling QAH to even larger models, exploring alternative teacher‑student architectures, and integrating QAH with other efficiency techniques such as sparsity or mixture‑of‑experts.

Full technical details, including the chunked KL implementation and distributed‑training findings, are available in the paper: https://huggingface.co/papers/2608.20953.

---