Text Generation Inference (TGI) Intel Gaudi Integration

Hugging Face has natively integrated Intel Gaudi hardware support into Text Generation Inference (TGI), enabling the deployment of Large Language Models (LLMs) on Intel's specialized AI accelerators using a production-ready serving stack. This integration removes the need for a separate fork and allows Gaudi users to access the latest TGI features immediately.

Unified Backend Architecture

TGI now supports Intel Gaudi directly within its main codebase (via PR #3091), replacing the previous requirement to use a separate tgi-gaudi fork. This transition was made possible by the new TGI multi-backend architecture, which streamlines the user experience and ensures that Gaudi hardware is treated as a first-class citizen in the TGI ecosystem.

Supported Intel Gaudi Hardware

The integration supports the full line of Intel Gaudi accelerators, including:

  • Gaudi1: Available via AWS EC2 DL1 instances.
  • Gaudi2: Available via Intel Tiber AI Cloud and Denvr Dataworks.
  • Gaudi3: Available via Intel Tiber AI Cloud, IBM Cloud, and OEMs including Dell, HP, and Supermicro.

Key Capabilities and Production Benefits

The Gaudi backend brings TGI's production-grade features to Intel hardware, offering an alternative to traditional GPU deployments. Key benefits include:

  • Production-Ready Features: Support for dynamic batching and streamed responses.
  • Deployment Flexibility: Increased hardware diversity and compelling price-performance for specific workloads.
  • Advanced Technical Support: Capabilities for multi-card inference (sharding), vision-language models, and FP8 precision.

Optimized Model Support

Hugging Face has optimized the modeling code specifically for Intel Gaudi hardware to maximize performance for both single and multi-card configurations. The following models are currently optimized for Gaudi:

  • Llama Series: Llama 3.1 (8B and 70B), Llama 3.3 (70B), and Llama 3.2 Vision (11B).
  • Mistral/Mixtral: Mistral (7B) and Mixtral (8x7B).
  • Other Architectures: CodeLlama (13B), Falcon (180B), Qwen2 (72B), Starcoder/Starcoder2, Gemma (7B), Llava-v1.6-Mistral-7B, and Phi-2.

Additionally, FP8 quantization is supported via the Intel Neural Compressor (INC) to further enhance performance.

Deployment and Implementation

Users can deploy TGI on Gaudi using the official Docker image (ghcr.io/huggingface/text-generation-inference:3.2.1-gaudi). Deployment requires a machine with Gaudi hardware and the --runtime=habana flag.

Future updates are planned to include support for additional models, including DeepSeek-r1/v3 and QWen-VL.

Sources