Hugging Face Inference Endpoints Case Study
Overview
Hugging Face has transitioned several of its CPU-based machine learning models from a self-managed AWS infrastructure to Hugging Face Inference Endpoints. This shift was driven by the need to reduce cognitive load and deployment time, resulting in lower latency and a simplified workflow for deploying models from the Hugging Face Hub.
Workflow Simplification
Moving to Inference Endpoints significantly reduces the number of steps required to move a model from training to production.
Previous Workflow (AWS ECS + Fargate)
Previously, the process involved a complex chain of manual steps:
- Train model on a GPU instance via CML and transformers.
- Upload the model to the Hugging Face Hub.
- Build a custom API to serve the model using FastAPI.
- Wrap the API in a Docker container.
- Upload the container to AWS Elastic Container Repository (ECR).
- Deploy the model to an AWS Elastic Container Service (ECS) cluster.
Current Workflow (Inference Endpoints)
The new managed process streamlines these steps into three:
- Train model on a GPU instance via CML and transformers.
- Upload the model to the Hugging Face Hub.
- Deploy using Hugging Face Inference Endpoints.
Performance and Latency Benchmarks
Testing conducted on a text classification model fine-tuned on RoBERTa (deployed in the eu-east-1 region) demonstrates that Inference Endpoints provide superior latency compared to the previous bespoke ECS setup.
| Instance Size | vCPU (cores) | Memory (GB) | ECS Latency (ms) | Inference Endpoints Latency (ms) |
|---|---|---|---|---|
| Small | 1 | 2 | - | ~296 |
| Medium | 2 | 4 | - | 156 ± 51 |
| Large | 4 | 8 | ~200 | 80 ± 30 |
| XLarge | 8 | 16 | - | 43 ± 31 |
For the "Large" instance size, the vanilla Hugging Face container was more than twice as fast as the bespoke container run on ECS, with the slowest response recorded at 108ms.
Cost Analysis
While Inference Endpoints are more expensive than the previous AWS Fargate setup, the cost increase is considered an acceptable trade-off for the reduction in MLOps overhead.
| Instance Size | vCPU | Memory (GB) | ECS Cost | Inference Endpoints Cost | % Difference |
|---|---|---|---|---|---|
| Small | 1 | 2 | $33.18 | $43.80 | 24% |
| Medium | 2 | 4 | $60.38 | $87.61 | 31% |
| Large | 4 | 8 | $114.78 | $175.22 | 34% |
| XLarge | 8 | 16 | $223.59 | $350.44 | 50% |
For a large CPU instance, the monthly cost difference is approximately $60. Hugging Face notes that for organizations deploying hundreds of ML microservices, the cost difference might warrant a different approach, but for their current scale, the time savings outweigh the financial cost.
Deployment and Operational Considerations
Deployment Methods
Users can deploy Inference Endpoints via the GUI, a RESTful API, or the hugie command-line tool, which allows for one-line configuration deployments (e.g., hugie endpoint create example/development.json).
Infrastructure as Code
As of the publication date, a custom Terraform provider for Inference Endpoints is not available, which is noted as a missing feature for those wishing to use Terraform state machines to track deployments.
Multi-Model Hosting
It is possible to host multiple models on a single endpoint by writing a custom Endpoint Handler class. While this has been demonstrated for GPU inference, it is assumed to work for CPU instances to optimize costs by maximizing memory utilization.