localai-org/apex-quant

Adaptive Precision for EXpert Models: MoE-aware mixed-precision quantization

What it solves

APEX addresses the inefficiency of uniform quantization in Mixture-of-Experts (MoE) models. Standard quantization often treats all weights equally, leading to either excessive size or unnecessary precision loss. APEX reduces model size and increases inference speed without sacrificing accuracy, allowing large MoE models to run on consumer-grade hardware (like 16GB VRAM GPUs) while maintaining or exceeding the quality of full-precision models.

How it works

APEX uses a mixed-precision strategy specifically designed for the structural properties of MoE models:

  • Tensor Classification: It categorizes weights into routed experts (which are sparse and can be compressed more), shared experts (which are critical and require high precision), and attention/SSM weights.
  • Layer-wise Precision Gradient: It assigns higher precision to the "edge" layers (first and last few layers) which are more sensitive to noise, and lower precision to the redundant middle layers.
  • Diverse Calibration: The "I-variants" use a calibration dataset covering chat, code, and reasoning instead of just Wikipedia, improving real-world task performance.
  • llama.cpp Integration: It leverages the --tensor-type-file and --tensor-type flags in stock llama.cpp to apply these specific precision assignments without requiring custom code changes.

Who it’s for

Developers and AI researchers who want to deploy large MoE models on limited hardware, specifically those using the GGUF format and llama.cpp ecosystem.

Highlights

  • High Efficiency: Beats Q8_0 perplexity at half the size and outperforms Unsloth Dynamic quantizations in speed and size.
  • Five Precision Tiers: Offers configurations ranging from "I-Quality" (maximum accuracy) to "Mini" (smallest viable size for 16GB VRAM).
  • Lossless Compression: Matches or beats full Q8_0 quality in some metrics while being significantly smaller.
  • Hardware Accessible: Enables running 35B MoE models on consumer GPUs like the RTX 4060 Ti 16GB.
  • Zero Code Changes: Works with standard llama.cpp and LocalAI.

Related

  • Project
    ModelCloud/GPTQModel

    A comprehensive LLM quantization toolkit that compresses models using methods like GPTQ and AWQ with hardware acceleration for NVIDIA, AMD, Intel, and Huawei NPUs.

  • Project
    microsoft/Tutel

    Tutel is Microsoft’s open‑source, high‑performance MoE library for large language models. It provides dynamic parallelism, low‑precision inference (NVFP4, MXFP4, FP8, BF16), support for up to 1 M‑token contexts, vision extensions, and Docker images that serve models like DeepSeek‑V3.2, Kimi‑K3, GLM‑5.x, Qwen‑3, and GPT‑OSS on both NVIDIA and AMD GPUs. Install via pip from the Git repo or build from source, then run the supplied Docker containers or use the Python API for custom MoE routing.

  • Project
    FujitsuResearch/OneCompression

    A Python package for automated LLM compression that uses advanced quantization techniques like QEP and AutoBit to reduce model size and VRAM usage.

  • Project
    kvcache-ai/ktransformers

    KTransformers is an open‑source framework for high‑performance, CPU‑GPU hybrid inference and fine‑tuning of large (especially Mixture‑of‑Experts) language models. It offers Intel‑optimized kernels, extensive quantization support, multi‑hardware compatibility, and tight integration with LlamaFactory for fast LoRA/full‑parameter training.

  • Dispatch
    Mixture of Experts (MoEs) in Transformers

    Hugging Face has redesigned the transformers library to make Mixture of Experts (MoEs) first-class citizens through a new weight loading refactor, a pluggable expert backend, and native expert parallelism.