Hugging Face Transformers timm Integration

Hugging Face has introduced TimmWrapper, a tool that enables any model from the PyTorch Image Models (timm) library to be used directly within the 🤗 transformers ecosystem. This integration allows users to leverage timm's extensive collection of computer vision models while utilizing transformers high-level APIs for inference, quantization, and fine-tuning.

Seamless Integration via TimmWrapper

The TimmWrapper bridges the gap between the timm library and the transformers library, making timm models compatible with standard Hugging Face workflows. This integration provides several key technical advantages:

  • Pipeline API Support: timm models can be plugged into the high-level transformers pipeline for streamlined inference.
  • Auto Class Compatibility: Models can be loaded using AutoModelForImageClassification and AutoImageProcessor, abstracting the complexity of model and processor loading.
  • Trainer API Integration: Users can fine-tune timm models using the standard Trainer API, maintaining a consistent workflow across different model architectures.
  • Round-Trip Compatibility: Models fine-tuned within the transformers ecosystem can be loaded back into timm using timm.create_model('hf-hub:my_org/my_fine_tuned_model', pretrained=True).

Optimized Inference and Deployment

The integration enables the use of advanced optimization techniques from the transformers ecosystem on timm models:

Quantization with bitsandbytes

Users can quantize any timm model for efficient inference using BitsAndBytesConfig. In a provided example using a ViT base model, 8-bit quantization reduced the model size from 346.27 MB to 88.20 MB (a 74.53% reduction) while maintaining nearly identical accuracy (0.33% vs 0.35% for a specific label).

Acceleration with torch.compile

The timm integration is fully compatible with torch.compile (introduced in PyTorch 2.0), allowing users to achieve faster inference times by compiling the model with a single line of code.

Flexible Fine-Tuning Options

TimmWrapper supports both standard and parameter-efficient fine-tuning (PEFT) methods:

Standard Fine-Tuning

timm models can be fine-tuned on custom datasets using the Trainer class, which manages the training loop, logging, and evaluation. This mirrors the exact workflow used for native transformers models.

LoRA (Low-Rank Adaptation)

Through the PEFT library, users can apply LoRA to timm models to train only a small fraction of parameters. In one example, a ViT model was fine-tuned with only 0.77% of its parameters being trainable (667,493 trainable parameters out of 86,543,818 total parameters). This allows for efficient training on consumer-grade hardware.

Practical Implementation Examples

  • Image Classification: The pipeline API can be used to load models like mobilenetv4_conv_medium (which lacks a native transformers implementation) for immediate inference.
  • Interactive Demos: The integration works with Gradio, allowing developers to build food classifier web apps using fine-tuned timm ViT models.
  • Model Loading: AutoImageProcessor and AutoModelForImageClassification can be used to load timm checkpoints directly from the Hugging Face Hub.

Sources