Accelerating Protein Language Model ProtST on Intel Gaudi 2

Intel and MILA have optimized the ProtST multi-modal protein language model for Intel Gaudi 2 accelerators, demonstrating significant speedups in both inference and fine-tuning compared to NVIDIA A100 GPUs. This integration is facilitated by the Optimum for Intel Gaudi open-source library, allowing users to port transformers-based scripts to Gaudi 2 with minimal code changes.

ProtST Model Overview

ProtST is a multi-modal language model for protein design based on text prompts, originally released by MILA and Intel Labs at ICML 2023. The model is designed for tasks such as predicting the subcellular location of an amino acid sequence—where the model identifies whether a sequence belongs to the nucleus, cell membrane, cytoplasm, or mitochondria.

Key capabilities include:

  • Zero-shot Performance: The ProtST-ESM-1b variant outperforms state-of-the-art few-shot classifiers in subcellular location prediction out of the box.
  • Accessibility: The model and associated datasets have been re-architected and shared on the Hugging Face Hub via the mila-intel organization.

Inference Performance on Intel Gaudi 2

Inference tests conducted on the ProtST-SubcellularLocalization dataset (containing 2,772 amino acid sequences with lengths between 79 and 1,999) show that Intel Gaudi 2 provides a substantial performance boost over the NVIDIA A100 80GB PCIe.

When running the model in full bfloat16 precision with a batch size of 1, the results are as follows:

  • Accuracy: Both the NVIDIA A100 and Intel Gaudi 2 achieved an identical accuracy of 0.44.
  • Speed: Intel Gaudi 2 delivered 1.76x faster inferencing speed than the A100.

Fine-tuning and Scaling Capabilities

Fine-tuning ProtST on downstream tasks can improve modeling accuracy. In a binary location task—determining if a protein is membrane-bound or soluble—the ProtST-ESM1b-for-sequential-classification model was fine-tuned in bfloat16 precision using the ProtST-BinaryLocalization dataset.

Training Accuracy and Speed

Fine-tuning results closely match the original research paper's findings, achieving approximately 92.5% accuracy across different hardware setups.

Hardware Comparison

Comparing a single Gaudi 2 accelerator to a single NVIDIA A100:

  • Fine-tuning Speed: A single Gaudi 2 is 2.92x faster than a single A100.
  • Scalability: Distributed training on Gaudi 2 scales near-linearly when increasing the number of accelerators to 4 or 8 units.

Implementation Resources

Developers can deploy ProtST on Intel Gaudi 2 using the following tools:

  • Optimum for Intel Gaudi: An open-source library that simplifies the porting of transformers-based scripts to Gaudi 2 hardware.
  • Intel Gaudi Documentation: Official technical guides for hardware implementation.

Sources