Hugging Face Integration for AMD Instinct MI300 GPU

Hugging Face and AMD have integrated the AMD Instinct MI300 GPU into the Hugging Face platform, providing seamless support across transformers, text-generation-inference (TGI), and other core libraries. This integration allows users to deploy models on AMD MI300 hardware, including Azure ND MI300x V5 VMs, without requiring code changes.

Open-Source and Production Enablement

To ensure long-term stability and correctness, Hugging Face has implemented a robust CI/CD pipeline for the MI300 line-up.

CI/CD Infrastructure

By leveraging Azure ND MI300x V5 VMs and a managed Kubernetes cluster, Hugging Face now runs tens of thousands of unit tests regularly on both the MI300 and the previous generation MI250 GPUs. This infrastructure abstracts hardware-specific pods from developers, ensuring that transformers and text-generation-inference remain compatible and performant across different AMD hardware platforms.

Production AI Workload Performance

Integration efforts focused on the modeling layer of TGI to optimize the execution of models like the Meta Llama family on MI300 hardware.

Inferencing Performance

Optimization efforts included the integration of Flash Attention v2, Paged Attention, GPTQ/AWQ compression techniques, and optimized fused kernels.

The Role of TunableOp

Starting with PyTorch 2.3, Hugging Face integrated AMD TunableOp, a mechanism that identifies the most efficient way to execute general matrix-multiplication (GEMM) based on shapes and data types. During the TGI warmup phase, TunableOp identifies optimal setups for the user's specific sequence length and batch size, then locks those routines for the duration of the server's operation. This results in an 8-10% speedup in latency for small input sequences during the autoregressive decoding phase.

Benchmarks: MI300 vs. MI250

Using Meta Llama 3 70B on Azure ND MI300x V5, the MI300X demonstrated significant gains over the MI250:

  • Time to First Token (Prefill): 2x-3x speedup.
  • Autoregressive Decoding Latency: 2x speedup.

Model Fine-Tuning Performance

Fine-tuning was tested using Transformers, PEFT (with LoRA), and DeepSpeed Zero3 via the Accelerate library on Llama 3 70B.

  • Training Speed: Training is approximately 2x faster on Azure VMs powered by MI300X compared to MI250 HPC servers.
  • Memory Capacity: The MI300X's 192 GB HBM3 memory allows the full loading and fine-tuning of Meta Llama 3 70B (approx. 140 GB in float16/bfloat16) on a single device, a task that is impossible on the 128 GB MI250.

Future Roadmap

Hugging Face is currently investing in "minifloat" support (float8 and lower). This is expected to reduce the size of the key-value cache in LLM inference by half. Future goals include combining float8 stored key-value caches with float8/float8 matrix-multiplications to further reduce memory footprints and increase performance.

Sources