microsoft/microxcaling

What it solves

This library allows researchers and data scientists to emulate MX-compatible data formats and bfloat quantization within PyTorch. It enables the exploration of how different low-precision numerical formats (like FP8, FP4, and INT8) affect the performance and accuracy of Deep Neural Networks (DNNs) without requiring specialized hardware that natively supports these formats.

How it works

The library simulates low-precision formats by performing computations in higher precision (float32, bfloat16, or fp16) while restricting the values to the range and precision of the target MX or bfloat formats. It provides drop-in replacements for standard PyTorch modules and functions, such as torch.matmul, torch.linear, and torch.nn.LayerNorm.

To improve simulation speed and numerical accuracy over native PyTorch GPU operations, the project includes custom CUDA extensions for quantization.

Who it’s for

It is designed for data scientists and AI researchers focusing on quantization and numerical precision exploration in DNNs.

Highlights

• Broad Format Support: Supports various MX-compatible formats including FP8 (e4m3, e5m2), FP4 (e2m1), and INT8.
• Flexible Configuration: Uses an mx_specs dictionary to configure scale bits, element formats for weights and activations, and block sizes.
• Seamless Integration: Offers two integration paths: manual replacement of PyTorch modules or automatic injection of operations via mx_mapping.inject_pyt_ops.
• High Performance: Includes custom CUDA kernels to avoid known PyTorch GPU numerical inaccuracies and increase simulation speed.
• Comprehensive Coverage: Covers both forward and backward pass quantization, as well as elementwise operations like GELU, Softmax, and LayerNorm.