Accelerating Vision-Language Models: BridgeTower on Habana Gaudi2

BridgeTower Model Overview

BridgeTower introduces multiple bridge layers that connect the top layers of uni‑modal encoders to each layer of a cross‑modal encoder, enabling effective bottom‑up cross‑modal alignment and fusion at different semantic levels. Pre‑trained on only 4M images from Conceptual Captions, SBU Captions, MSCOCO Captions and Visual Genome, it achieves state‑of‑the‑art performance on vision‑language tasks such as 78.73% accuracy on VQAv2 test‑std, outperforming METER by 1.09% with negligible extra parameters and compute.

Hardware Platforms

Habana Gaudi2 provides eight HPUs, each with 96 GB of memory, and is supported by Optimum Habana for easy porting of Transformers scripts. Nvidia H100 is the latest GPU generation with a dedicated Transformer Engine for fp8 mixed‑precision and 80 GB of memory. Nvidia A100 (80 GB variant) uses the third generation of Tensor Core technology and remains the fastest GPU widely available in cloud providers.

Benchmark Setup

We fine‑tuned the BridgeTower Large checkpoint (866 M parameters) on the New Yorker Caption Contest dataset, keeping hyper‑parameters identical across all accelerators and measuring training throughput in samples per second.

Impact of dataloader_num_workers

Allocating more subprocesses for data loading improves throughput on all devices. With a batch size of 48 per device we observed:

  • Gaudi2 HPU: 601.5 samples/s (workers=0), 747.4 samples/s (workers=1), 768.7 samples/s (workers=2).
  • H100 GPU: 336.5 samples/s (workers=0), 580.1 samples/s (workers=1), 602.1 samples/s (workers=2).
  • A100 GPU: 227.5 samples/s (workers=0), 339.7 samples/s (workers=1), 345.4 samples/s (workers=2). These results show Gaudi2 is x1.28 faster than H100 and x2.23 faster than A100 when using two workers, and that increasing workers yields speedups of x1.28 on Gaudi2, x1.79 on H100 and x1.52 on A100.

Impact of hardware‑accelerated data loading with Optimum Habana

Moving image decoding and augmentations to the accelerator via the --mediapipe_dataloader flag (available only on Gaudi2) further increases throughput. The same experiments with two dataloader workers give:

  • Gaudi2 HPU: 847.7 samples/s (workers=2 + mediapipe_dataloader).
  • H100 and A100: not applicable (the flag does not work on these GPUs). Compared to the base run (workers=0) this represents an x1.41 speedup on Gaudi2, making it x1.41 faster than H100 and x2.45 faster than A100 when using two workers and the media pipeline.

Reproducing this benchmark

To reproduce the results, access Gaudi2 through Intel Developer Cloud, install the latest Optimum Habana, clone the repository, install requirements, and run the provided run_bridgetower.py script with the appropriate arguments. For Gaudi2 the base command includes --use_habana --use_lazy_mode --use_hpu_graphs_for_inference --gaudi_config_name Habana/clip. Adding --dataloader_num_workers N and --mediapipe_dataloader tests other configurations. For A100 and H100 replace Habana‑specific classes with the standard Transformers Trainer and TrainingArguments, remove Gaudi config references, and import set_seed from Transformers.

Conclusion

When training vision‑language models, two simple techniques—adding more dataloader workers and offloading image decoding/augmentation to the accelerator—deliver substantial speedups. Using Optimum Habana on Habana Gaudi2 yields about 1.4× the speed of Nvidia H100 and 2.5× the speed of Nvidia A100 80GB when fine‑tuning BridgeTower, requiring only a few extra training arguments.

Sources