Hugging Face and AMD GPU Acceleration for LLMs

Hugging Face and AMD have partnered to provide out-of-the-box acceleration for Large Language Models (LLMs) on AMD Instinct GPUs. This integration allows users to run Hugging Face Transformers models on AMD hardware without requiring any code changes, significantly expanding the hardware options for high-performance AI inference and training.

Zero-Code Integration for AMD Instinct GPUs

All Hugging Face Transformers models and tasks are now supported on AMD Instinct GPUs. Users can deploy models using the same PyTorch code typically used for NVIDIA GPUs, simply by targeting the "cuda" device.

To ensure stability and performance, Hugging Face has implemented integrated testing of open-source libraries on AMD Instinct GPUs within their datacenters, utilizing servers deployed in Iceland via Verne Global to minimize carbon impact.

Technical Optimizations and Feature Support

Beyond native support, the collaboration has integrated several state-of-the-art acceleration tools and kernels for ROCm-powered GPUs:

  • Flash Attention v2: Integrated natively into Transformers and Text Generation Inference (TGI) via AMD open-source efforts.
  • Paged Attention: Supported through vLLM and various fused kernels in TGI for ROCm.
  • DeepSpeed: Officially validated and supported for ROCm-powered GPUs using Transformers.
  • GPTQ: Weight compression for reducing memory requirements is supported via direct integration with AutoGPTQ and Transformers.
  • ONNX Runtime: Support for ONNX model execution on ROCm GPUs via the ROCMExecutionProvider using the Optimum library.
  • Optimum-Benchmark: A utility provided to benchmark Transformers performance on AMD GPUs across normal and distributed settings.

Hardware Performance: AMD Instinct MI250 vs. NVIDIA A100

The AMD Instinct MI250 features 128 GB of High Bandwidth Memory, split into two distinct ROCm devices (64 GB each). This architecture allows a single MI250 card to function as two PyTorch devices, facilitating tensor and data parallelism.

Benchmarks conducted using optimum-benchmark reveal the following performance advantages of the MI250 over the A100:

  • Decoding Throughput: In production settings with large batches, the MI250 delivers more than 2.33x more tokens per second.
  • Prefill Latency: The MI250 reduces the time to the first token (prefill latency) by half compared to the A100.
  • Training: The MI250 supports larger batch sizes and achieves higher training throughput.

Production Deployment with Text Generation Inference (TGI)

Text Generation Inference (TGI) now provides initial support for AMD Instinct MI210 and MI250 GPUs. TGI leverages the optimizations mentioned above to provide an end-to-end deployment solution for LLMs at scale.

Performance benchmarks for the Llama model family demonstrate the MI250's memory advantage. For Llama 70B (which requires 138 GB in float16), the MI250's 128 GB global memory allows for larger workloads, longer sequences, and larger batches compared to the A100's 80 GB, which encountered out-of-memory errors in similar configurations.

To optimize matrix multiplication, Hugging Face has utilized the AMD GeMM Tuner tool, which is expected to be released as part of PyTorch in a future update.

Production deployment is available via the Docker image: ghcr.io/huggingface/text-generation-inference:1.2-rocm.

Future Roadmap

Hugging Face and AMD are expanding their support across three primary areas:

  1. Consumer Hardware: Bringing validation and support to AMD Radeon GPUs for local desktop usage.
  2. Next-Gen Server GPUs: Optimizing performance for the upcoming AMD Instinct MI300 lineup.
  3. Edge AI: Enhancing support for AMD Ryzen AI technology in laptop CPUs to enable private, on-device AI execution. Ryzen AI compatible models are currently available on the Hugging Face Hub.

Sources