Transformers 4.40 performance upgrades for OpenAI GPT‑OSS: zero‑build kernels, MXFP4 quantization, parallelism, and faster loading

TL;DR

OpenAI’s GPT‑OSS models are now fully supported in the Hugging Face transformers library with a suite of performance upgrades—including zero‑build downloadable kernels, MXFP4 4‑bit quantization, tensor and expert parallelism, dynamic sliding‑window cache, continuous batching, and faster model loading—so developers can load, run, and fine‑tune these models more efficiently on a single GPU or across multiple GPUs.

Zero‑Build Kernels from the Hub

Conclusion: Pre‑compiled custom kernels can be downloaded automatically, eliminating build‑time dependencies and delivering up to 10× speedups for common LLM operations.

transformers now integrates the kernels package, which fetches binary kernels (e.g., Liger RMSNorm, MegaBlocks MoE, FlashAttention 3) from the Hugging Face Hub on first use. Users enable them by passing use_kernels=True when loading a model:

from transformers import AutoTokenizer, AutoModelForCausalLM
import logging
logging.basicConfig(level=logging.INFO)

model_id = "openai/gpt-oss-20b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
    use_kernels=True,
)

Log output confirms which kernels are loaded, e.g., LigerRMSNorm and MegaBlocksMoeMLP. Benchmarks in the original post show these kernels excel for larger batch sizes, though users should benchmark for their own workloads.

FlashAttention 3 with Attention Sinks

Conclusion: Enabling the FlashAttention 3 kernel that supports attention sinks yields lower latency on Hopper‑class GPUs.

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
    attn_implementation="kernels-community/vllm-flash-attn3",
)

The kernel is compatible with NVIDIA Hopper GPUs and leverages the attention‑sink feature introduced in GPT‑OSS.

MXFP4 4‑Bit Quantization

Conclusion: MXFP4 reduces VRAM usage by up to 80 GB for the 120‑B parameter GPT‑OSS model, making it runnable on a single consumer GPU while preserving inference speed via specialized Triton kernels.

What MXFP4 Is

MXFP4 uses an E2M1 4‑bit floating format (1 sign, 2 exponent, 1 mantissa) combined with blockwise scaling (32‑element blocks share a scale). This design retains dynamic range despite the coarse mantissa.

Native Support in transformers

The library ships a quantizer (quantizer_mxfp4.py) and integration hooks (integrations/mxfp4.py). When a model’s config contains "quant_method": "mxfp4", the MXFP4 pathway is automatically selected.

from transformers import GptOssConfig
cfg = GptOssConfig.from_pretrained("openai/gpt-oss-120b")
print(cfg.quantization_config)

If the required environment is present—accelerate, kernels, triton>=3.4, and a GPU with compute capability ≥ 7.5—the model runs in MXFP4 mode; otherwise it falls back to bfloat16, consuming roughly four times more memory.

Memory Savings

Figure 3 in the source post visualizes VRAM consumption: the quantized 20 B model occupies ~16 GB versus ~64 GB when de‑quantized. The same scaling holds for the 120 B model (≈80 GB vs. >300 GB).

Tensor Parallelism (TP)

Conclusion: TP shards tensors across GPUs, enabling models that exceed a single‑GPU memory budget to achieve higher throughput on multi‑GPU nodes.

transformers now accepts a tp_plan="auto" argument in from_pretrained, which selects a built‑in sharding recipe. Example usage:

from transformers import PreTrainedTokenizerFast, GptOssForCausalLM
model_id = "openai/gpt-oss-120b"
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_id)
model = GptOssForCausalLM.from_pretrained(
    model_id,
    tp_plan="auto",
    dtype="auto",
).eval()

TP works best on a single node with fast intra‑node links and is distinct from device_map="auto", which only handles memory placement.

Expert Parallelism (EP)

Conclusion: EP distributes MoE experts across GPUs, complementing TP and further reducing per‑GPU compute load for mixture‑of‑experts models.

Enable EP via the DistributedConfig:

from transformers import DistributedConfig
model = GptOssForCausalLM.from_pretrained(
    model_id,
    distributed_config=DistributedConfig(enable_expert_parallel=True),
    dtype="auto",
).eval()

When EP is active, TP is automatically enabled, giving combined benefits.

Dynamic Sliding‑Window Layer & Cache

Conclusion: The new DynamicSlidingWindowLayer and DynamicCache stop KV‑cache growth after the attention window is reached, halving cache memory for hybrid‑attention models like GPT‑OSS.

The feature is enabled by default; developers can explicitly create a cache:

from transformers import AutoModelForCausalLM, AutoTokenizer, DynamicCache
model = AutoModelForCausalLM.from_pretrained(
    "openai/gpt-oss-20b",
    dtype="auto",
    device_map="auto",
).eval()
cache = DynamicCache(config=model.config)

Benchmarks (Figure 6) show substantial memory reduction and latency improvements for long generations.

Continuous Batching & Paged Attention

Conclusion: generate_batch implements dynamic (continuous) batching, keeping GPUs busy by refilling finished slots with new requests, yielding higher tokens‑per‑second than static batching.

The API is experimental and intended for research/evaluation rather than production serving (where vLLM or SGLang excel). The source post provides a reference script and benchmark showing up to ~2× speedup over static batching.

Faster Model Loading

Conclusion: transformers now pre‑allocates a large memory block per GPU before copying weights, cutting the thousands of tiny allocation calls that previously slowed loading of multi‑billion‑parameter models.

This behavior is automatic when using device_map="auto" or any explicit device map, and also benefits TP‑enabled runs.

Overall Impact

Conclusion: By integrating community‑driven kernels, MXFP4 quantization, and advanced parallelism strategies directly into transformers, Hugging Face dramatically lowers the hardware barrier for running state‑of‑the‑art LLMs, speeds up both inference and fine‑tuning, and provides a unified reference implementation for other toolkits (MLX, llama.cpp, vLLM).

Developers can now:

  • Load GPT‑OSS 20 B on a free‑tier Colab GPU using MXFP4.
  • Scale GPT‑OSS 120 B across 4 GPUs with a single torchrun command.
  • Benefit from automatic kernel downloads without manual compilation.
  • Reduce KV‑cache memory for long‑context applications.

All of these advances are released as open‑source code in the transformers repository, with detailed PR references and example scripts linked throughout the post.

Key Resources

How to Get Started

  1. Install the latest transformers (≥ 4.40) with optional extras:
    pip install "transformers[torch,accelerate,triton,kernels]"
  2. Choose a model (e.g., openai/gpt-oss-20b).
  3. Enable the desired features (use_kernels=True, tp_plan="auto", DistributedConfig(enable_expert_parallel=True)).
  4. Run the provided benchmark scripts to verify speed and memory gains on your hardware.

By following these steps, practitioners can immediately reap the performance benefits announced in the September 2025 Hugging Face blog post.

Future Directions The blog emphasizes that these integrations are a snapshot; the library will continue to evolve with community contributions, adding support for newer quantization formats, more kernel backends, and tighter coupling with serving stacks like vLLM.

Sources