Vision Transformers on Hugging Face Optimum Graphcore

Hugging Face and Graphcore have integrated Vision Transformer (ViT) models into the Hugging Face Optimum Graphcore library, allowing users to fine-tune these models on Graphcore Intelligence Processing Units (IPUs). This integration simplifies the deployment of high-performance computer vision models by combining pre-trained ViT checkpoints with IPU-specific hardware acceleration.

Vision Transformer (ViT) Architecture

Vision Transformers (ViT) apply the self-attention mechanism—originally developed for natural language processing (NLP) models like BERT and GPT—to image recognition. Unlike Convolutional Neural Networks (CNNs) that use pixel arrays, ViT models divide an input image into small patches (visual tokens). Each patch is linearly encoded into a vector representation that the transformer processes individually.

Key characteristics of the ViT approach include:

  • Feature Extraction: Pre-training allows ViT to learn inner representations of images. A linear layer can be added on top of the pre-trained visual encoder (typically the [CLS] token) to perform downstream classification tasks.
  • Performance: ViT models can achieve higher recognition accuracy with lower computational costs compared to CNNs.
  • Applications: ViT is used across various domains, including object detection, segmentation, and healthcare diagnostics (e.g., detecting COVID-19, breast cancer, and Alzheimer’s disease).

Optimizing ViT for Graphcore IPUs

Graphcore IPUs are designed for the massively parallel nature of ViT models. The hardware utilizes a MIMD (Multiple Instruction, Multiple Data) architecture and the IPU-Fabric scale-out solution to parallelize training.

Technical optimizations provided by the Optimum Graphcore library include:

  • Parallelism: The use of pipeline parallelism increases the batch size per data parallelism instance, improves memory access efficiency, and reduces parameter aggregation communication time.
  • Ready-to-use Checkpoints: Graphcore provides IPU-trained model checkpoints and configuration files (such as Graphcore/vit-base-ipu) to eliminate the need for users to train ViT models from scratch, which typically requires massive datasets.
  • Seamless Integration: Users can leverage existing checkpoints from the Hugging Face Model Hub, such as google/vit-base-patch16-224-in21k, and apply IPU-specific configurations via the IPUConfig and IPUTrainer classes.

Case Study: Multi-Label Chest X-Ray Classification

To demonstrate the efficiency of the Optimum Graphcore workflow, a ViT model was fine-tuned on the ChestX-ray14 dataset, which contains 112,120 frontal view X-rays from 30,805 patients.

Dataset Preparation

Because X-ray images can exhibit multiple diseases simultaneously, the task is treated as a multi-label classification problem. The workflow involves:

  1. Label Encoding: Transforming text labels into N-hot encoded arrays (booleans) to represent multiple concurrent diseases.
  2. Preprocessing: Using AutoImageProcessor to resize images to 224x224, convert grayscale images to RGB, and normalize channels with a mean and standard deviation of 0.5.
  3. Data Loading: Utilizing the Arrow file format via Hugging Face Datasets for fast loading during training.

Training and Evaluation

  • Model: The google/vit-base-patch16-224-in21k checkpoint (pre-trained on 14 million ImageNet-21k images) was used.
  • Trainer: The IPUTrainer class was employed, which handles model compilation for the IPU and manages training and evaluation.
  • Metric: The Area Under the ROC Curve (AUC_ROC) was implemented as the primary performance metric, as it is insensitive to class imbalance and effectively measures the model's ability to separate different diseases.
  • Results: Training logs indicated a rapid reduction in loss, stabilizing around 0.1, with a learning rate following a 25% warm-up period and subsequent cosine decay.

Accessibility via Paperspace Gradient

Through a partnership with Paperspace, access to Hugging Face Optimum models powered by Graphcore IPUs is available via Gradient's web-based Jupyter notebooks. This allows developers to experiment with ViT, BERT, and RoBERTa on IPU hardware without requiring local infrastructure.

Sources