Hugging Face AWS Inferentia2 Integration

Hugging Face has integrated AWS Inferentia2 (Inf2) acceleration into its deployment ecosystem, allowing users to deploy over 100,000 models via Amazon SageMaker and providing direct Inf2 instance options within Hugging Face Inference Endpoints. This integration aims to provide high performance and cost-efficiency for production AI workloads using the optimum-neuron open-source library.

Expanded Model Support on Amazon SageMaker

AWS customers can now deploy over 100,000 public models from the Hugging Face Hub on AWS Inferentia2 instances using Amazon SageMaker. This expansion includes support for six machine learning tasks and 14 new model architectures:

  • Supported Architectures: albert, bert, camembert, convbert, deberta, deberta-v2, distilbert, electra, roberta, mobilebert, mpnet, vit, xlm, and xlm-roberta.
  • Supported Tasks: Text-classification, text-generation, token-classification, fill-mask, question-answering, and feature-extraction.

This deployment path allows users to leverage SageMaker's managed features for MLOps, governance, and model fine-tuning.

AWS Inferentia2 in Hugging Face Inference Endpoints

Hugging Face Inference Endpoints now includes AWS Inferentia2 instances as a configuration option, enabling one-click deployment of Hub models. For Large Language Models (LLMs) such as Llama 3, two specific instance flavors are available:

  • Inf2-small: 2 cores and 32 GB memory, priced at $0.75/hour (optimized for Llama 3 8B).
  • Inf2-xlarge: 24 cores and 384 GB memory, priced at $12/hour (optimized for Llama 3 70B).

Inference Endpoints are billed by the second, featuring automated replica autoscaling and a "scale to zero" capability to optimize costs.

Technical Implementation via Text Generation Inference (TGI)

For LLMs, Inference Endpoints utilizes Text Generation Inference for Neuron (TGI). TGI is designed for production-scale LLM serving and supports continuous batching and streaming. Furthermore, LLMs deployed via TGI are compatible with the OpenAI SDK Messages API, allowing developers to switch endpoints without modifying their existing application code.

Future Roadmap

Hugging Face is expanding the scope of models compatible with AWS Inferentia2 within Inference Endpoints. Planned updates include: n* New Model Types: Support for Diffusion and Embedding models to enable accelerated image generation and semantic search/recommendation systems.

  • Performance Optimization: Ongoing improvements to Text Generation Inference (TGI) on Neuronx to increase the efficiency and speed of LLM deployments.

Sources