Google Cloud TPUs on Hugging Face Inference Endpoints and Spaces

Hugging Face has integrated Google Cloud TPU v5e support into Inference Endpoints and Spaces, allowing AI developers to accelerate applications using Google's custom AI hardware. This integration provides a scalable, cost-effective alternative for deploying and serving Large Language Models (LLMs) and AI-powered demos.

TPU v5e Integration in Hugging Face Inference Endpoints

Google Cloud TPU v5e is now available as a hardware option for Hugging Face Inference Endpoints, enabling seamless deployment of Generative AI models on managed infrastructure. Users can access these TPUs by selecting Google Cloud Platform in the us-west1 region.

Available TPU Configurations and Pricing

Inference Endpoints offers three initial TPU v5e instance configurations:

  • v5litepod-1: 1 core, 16 GB memory ($1.375/hour)
  • v5litepod-4: 4 cores, 64 GB memory ($5.50/hour)
  • v5litepod-8: 8 cores, 128 GB memory ($11.00/hour)

For models with up to 2 billion parameters, the v5litepod-1 configuration is sufficient. For larger models, the v5litepod-4 or v5litepod-8 configurations are recommended to prevent memory budget issues and reduce latency.

Technical Implementation and Model Support

To facilitate the deployment of Hugging Face models on Google TPUs, the following technical components were developed:

  1. Optimum TPU: An open-source library created to simplify the training and deployment of Hugging Face models on Google TPUs.
  2. TGI Integration: Inference Endpoints utilizes Optimum TPU in conjunction with Text Generation Inference (TGI) to serve LLMs on TPU hardware.
  3. Supported Architectures: Initial support includes popular model architectures such as Gemma, Llama, and Mistral.

TPU v5e Support for Hugging Face Spaces

Developers can now upgrade Hugging Face Spaces to run on TPU v5e instances, allowing for the creation and hosting of high-performance AI-powered demos and applications. The hardware configurations and pricing for Spaces are identical to those available for Inference Endpoints:

  • v5litepod-1: 1 core, 16 GB memory ($1.375/hour)
  • v5litepod-4: 4 cores, 64 GB memory ($5.50/hour)
  • v5litepod-8: 1 core, 16 GB memory ($1.375/hour) — Correction based on source: The source lists v5litepod-8 as 8 cores and 128 GB memory ($11.00/hour) for both services.

Users can enable TPU support by navigating to the Settings menu within their specific Space and selecting the desired TPU configuration.

Sources