Hugging Face Transformers v5 release notes / what's new

Hugging Face has released Transformers v5 (v5.0.0rc-0), a major update focused on interoperability and simplification. The release transforms the library into a standardized "source of truth" for model definitions, enabling seamless movement between training, inference, and local deployment across the AI ecosystem.

Simplification and Model Standardization

Transformers v5 prioritizes clean, modular model definitions to ensure the library remains the industry standard for model architectures.

Modular Design and Model Additions

To reduce the maintenance burden and accelerate the integration of new architectures, Hugging Face has implemented a modular approach. This design significantly reduces the lines of code required for contributions and reviews. A key technical addition is the AttentionInterface, which centralizes attention methods (such as FA1/2/3, FlexAttention, and SDPA), while keeping the eager method within the modeling file.

Tooling for Automated Conversion

New machine learning-based tooling is being developed to identify similarities between independent modeling files and existing architectures. This allows the team to automate the conversion process by generating draft PRs for model integration, reducing manual effort and ensuring consistency.

Code Reduction and Backend Consolidation

Transformers v5 streamlines its codebase through several critical refactors:

  • PyTorch as Sole Backend: The library is sunsetting Flax and TensorFlow support to focus exclusively on PyTorch. Compatibility with the JAX ecosystem is being maintained through partnerships.
  • Tokenization Overhaul: The concept of "Fast" and "Slow" tokenizers is removed in favor of a unified tokenizers backend. Support for Sentencepiece and MistralCommon remains available as non-default alternatives.
  • Image Processing: Image processors now only exist in their fast variant, relying on the torchvision backend.

Enhanced Training Capabilities

While previous versions focused heavily on fine-tuning, v5 introduces significant improvements for pre-training and full-training at scale.

Pre-training at Scale

To support large-scale pre-training, Hugging Face reworked model initialization and optimized kernels for both forward and backward passes. The library now offers extended compatibility with pre-training tools including torchtitan, megatron, and nanotron.

Fine-tuning and Post-training

Transformers v5 maintains deep compatibility with PyTorch-based fine-tuning tools such as Unsloth, Axolotl, LlamaFactory, and TRL. It also ensures interoperability with JAX-based tools like MaxText.

Inference and Production Integration

Transformers v5 introduces new APIs and specialized kernels to bridge the gap between model definition and high-performance deployment.

New Inference APIs

Two primary additions enhance serving capabilities:

  • Continuous Batching and Paged Attention: Support for these mechanisms is now integrated to improve throughput.
  • transformers serve: A new serving system that deploys an OpenAI API-compatible server, specifically optimized for evaluation use-cases.

Ecosystem Interoperability

Rather than competing with specialized inference engines, Transformers v5 acts as a backend for them. This allows new models added to transformers to be immediately available in engines like vLLM and SGLang, utilizing their specific optimizations (e.g., dynamic batching and specialized kernels).

Additionally, the library improves interoperability with local inference tools:

  • GGUF Support: Users can now load GGUF files directly in transformers for fine-tuning or convert transformers models to GGUF for use with llama.cpp.
  • MLX Compatibility: Safetensors files from transformers are directly compatible with MLX models.
  • On-Device Deployment: Collaboration with the executorch team and optimum-executorch expands model availability on-device, including multimodal (vision and audio) models.

First-Class Quantization Support

Quantization is now a central focus of Transformers v5, moving from an add-on to a first-class citizen in the weight-loading process. This ensures full compatibility with major features and provides a reliable framework for both training and inference. This shift is supported by integrations with TorchAO and bitsandbytes, enabling better support for Tensor Parallelism (TP) and Mixture of Experts (MoE) in low-precision formats.

Sources