Fine-Tuning Vision Transformer (ViT) for Image Classification with Hugging Face Transformers

Overview

Hugging Face has detailed a workflow for fine-tuning the Vision Transformer (ViT) for image classification tasks. By treating image patches as tokens—similar to how words are treated in Natural Language Processing (NLP)—ViT allows transformer-based architectures to be applied to computer vision. This process involves splitting an image into a grid of sub-image patches, embedding each patch with a linear projection, and passing the resulting sequence of tokens to the transformer model.

Image Preprocessing with ViTImageProcessor

Correct image transformation is critical for ViT model performance because the model expects inputs to follow the specific normalization and resizing parameters used during its original training.

To ensure consistency, the ViTImageProcessor is used to load configurations from a pretrained model (such as google/vit-base-patch16-224-in21k). The processor handles the following operations:

  • Resizing: Adjusting the image to a standard size (e.g., 224x224 pixels).
  • Normalization: Applying specific mean and standard deviation values to the pixel data.
  • Resampling: Ensuring the image is correctly sampled for the model.

Processing an image through this tool returns a dictionary containing pixel_values, which is the numeric tensor representation required by the model.

Efficient Dataset Handling with Transforms

When working with image datasets, applying transformations to every example via ds.map can be computationally expensive and slow. Instead, Hugging Face recommends using the with_transform method from the datasets library.

Transforms are applied in real-time as examples are indexed, rather than pre-processing the entire dataset. This approach requires a transform function that can handle batches of data, converting a list of PIL images into the necessary pixel_values tensors while retaining the associated labels.

Fine-Tuning Pipeline and Configuration

Fine-tuning a ViT model involves several key components to ensure the model converges and evaluates correctly:

Data Collation and Metrics

Because batches are delivered as lists of dictionaries, a custom collate_fn is required to stack pixel_values and labels into torch tensors. For evaluation, the accuracy metric from the evaluate library is typically used to compare the model's predicted class (determined via np.argmax on the predictions) against the ground truth labels.

Model Initialization

To adapt a pretrained ViT for a specific task, ViTForImageClassification is used. The model is initialized with num_labels to create a classification head with the correct number of output units. Additionally, id2label and label2id mappings are provided to ensure that the model's outputs are human-readable when hosted on the Hugging Face Hub.

Training Arguments

Key configurations in TrainingArguments include:

  • remove_unused_columns=False: This is critical. By default, the trainer drops columns not used by the model's forward pass. However, since the image column is needed by the transform function to create pixel_values, it must be preserved.
  • fp16=True: Enables mixed-precision training to reduce memory usage and speed up training.
  • evaluation_strategy="steps": Allows for periodic evaluation during the training process.

Performance Results

In the provided example using the beans dataset (which classifies healthy vs. unhealthy bean leaves), the fine-tuned ViT model achieved an evaluation accuracy of 0.985 and an evaluation loss of 0.0637 after 4 epochs of training.

Sources