Habana Gaudi2 vs Nvidia A100 80GB Performance Benchmarks
Habana Gaudi2 delivers approximately twice the throughput of the Nvidia A100 80GB for both training and inference tasks. This performance gain is supported by increased memory capacity and seamless integration with the 🤗 Optimum Habana library.
Hardware Specifications and Compatibility
Gaudi2 is the second-generation AI hardware accelerator from Habana Labs. A single server typically contains eight accelerator devices, each equipped with 96GB of memory. This is a significant increase over the 80GB found in the Nvidia A100 80GB and the 32GB available in the first-generation Gaudi.
Because the Habana SDK, SynapseAI, is common to both first-generation Gaudi and Gaudi2, the 🤗 Optimum Habana interface remains identical for both hardware generations. Workflows designed for first-gen Gaudi are compatible with Gaudi2 without requiring code changes.
Training Performance: BERT Pre-training
Gaudi2 demonstrates substantial speedups in BERT pre-training compared to both first-generation Gaudi and Nvidia A100 80GB. When using a batch size of 32 samples per device, Gaudi2 achieved a throughput of 1580.2 samples/s, compared to 981.6 samples/s for the A100.
Key findings from BERT pre-training include:
- Speedup over First-gen Gaudi: Gaudi2 achieved a 3.04x speedup with a batch size of 32. By increasing the batch size to 64 (leveraging Gaudi2's larger memory), the total training time was reduced by a factor of 5.75, with throughput increasing to 1835.8 samples/s (a 3.53x increase over first-gen Gaudi).
- Speedup over Nvidia A100: Gaudi2 outperformed the A100 at both tested batch sizes (32 and 64), maintaining a throughput advantage consistent with Habana's announced 1.8x speedup for phase 1 of BERT pre-training.
| Hardware | Batch Size (per device) | Throughput (samples/s) | Speedup vs First-gen Gaudi |
|---|---|---|---|
| First-gen Gaudi | 32 | 520.2 | x1.0 |
| Gaudi2 | 32 | 1580.2 | x3.04 |
| Gaudi2 | 64 | 1835.8 | x3.53 |
| A100 | 32 | 981.6 | x1.89 |
| A100 | 64 | 1082.6 | x2.08 |
Inference Performance: Stable Diffusion
Gaudi2 significantly reduces latency for image generation using Stable Diffusion. For a batch size of 8 samples, Gaudi2 achieved a latency of 0.925s per image, making it 2.84x faster than the Nvidia A100 (2.63s per image) and 3.51x faster than first-generation Gaudi (3.25s per image).
These benchmarks were conducted using the Habana/stable-diffusion configuration and the 🤗 Optimum Habana 1.3 release, which introduced official support for Stable Diffusion. To ensure accuracy, the first two batches were discarded to account for model compilation times.
Large Model Fine-tuning: T5-3B
The 96GB memory per device on Gaudi2 allows for the fine-tuning of larger models that were impossible to run on first-generation Gaudi. In tests involving the T5-3B model (3 billion parameters) fine-tuned on the CNN DailyMail dataset for summarization, Gaudi2 outperformed the Nvidia A100 80GB.
Gaudi2 achieved a throughput of 19.7 samples/s compared to 8.07 samples/s for the A100, representing a 2.44x speedup. These runs were performed in fp32 with gradient checkpointing enabled on eight devices. The report notes that memory footprint optimization in future SynapseAI releases is expected to further improve these results.
Summary of Performance Gains
Across all tested workloads, Gaudi2 consistently provides higher throughput and lower latency than the Nvidia A100 80GB:
- BERT Pre-training: Up to 2.08x higher throughput than A100.
- Stable Diffusion Inference: 2.84x lower latency than A100.
- T5-3B Fine-tuning: 2.44x higher throughput than A100.