charlie12345/ROCmFPX

What it solves

ROCmFPX provides specialized GGUF model-weight quantization formats designed specifically for AMD hardware. It addresses the need for efficient model storage and faster inference on Radeon GPUs by offering a variety of bit-widths (from 2-bit to 8-bit) that are optimized for AMD's architecture, reducing memory traffic and increasing token throughput compared to generic formats.

How it works

The project adds experimental weight formats to llama.cpp, implementing accelerated kernels for HIP/ROCm and Vulkan backends, with CPU reference paths for correctness. It includes several quantization families:

• ROCmFP2/3/4/6/8: Specialized floating-point-like quants of varying precision.
• ROCmI4: A signed 4-bit integer format that can leverage the native v_wmma_i32_16x16x16_iu4 instruction on AMD Strix Halo (gfx1151) hardware for accelerated batched matrix multiplication.
• MTP Acceleration: Supports self-speculative decoding for models with an embedded MTP/NextN head, allowing the model to verify multiple proposed tokens in a single batch to increase decode speed.
• Agent-Aware Presets: Specific quantization recipes (e.g., *_COHERENT or *_AGENT) that protect critical tensors to maintain the quality of structured outputs like JSON and code.

Who it’s for

Users running LLMs on AMD Radeon GPUs, particularly those using llama.cpp and those with Strix Halo (gfx1151) hardware looking to maximize inference performance and minimize VRAM usage.

Highlights

• AMD-Optimized Quants: Native support for 2, 3, 4, 6, and 8-bit GGUF formats.
• Multi-Backend Support: Accelerated paths for both Vulkan and HIP/ROCm.
• Strix Halo Specialization: Opt-in IU4/W4A4 acceleration for gfx1151 devices.
• Speculative Decoding: Built-in MTP acceleration for faster token generation without needing a separate draft model.
• Precision Control: Offers a range of formats from the smallest (ROCmFP2) to high-quality references (ROCmFP8).