Deploy Hugging Face models easily with Amazon SageMaker

Hugging Face and Amazon SageMaker have introduced new Inference Deep Learning Containers (DLCs) and a dedicated Inference Toolkit, allowing users to deploy trained Transformer models or any of the 10,000+ models from the Hugging Face Model Hub to production-ready AWS endpoints with minimal code.

Simplified Model Deployment with Inference DLCs

Amazon SageMaker now supports Hugging Face Inference Deep Learning Containers (DLCs), which provide fully tested, optimized deep learning environments that eliminate the need for manual installation or configuration. These containers allow for the rapid creation of production-ready endpoints that scale within the AWS environment and include built-in monitoring and enterprise features.

Users can deploy models using the HuggingFaceModel class in the SageMaker Python SDK. The deployment process is streamlined such that a trained model can be deployed for inference with a single additional line of code.

The SageMaker Hugging Face Inference Toolkit

To further reduce deployment friction, Hugging Face created the SageMaker Hugging Face Inference Toolkit. This toolkit leverages pipelines from the transformers library to enable "zero-code" deployments, meaning users do not need to write custom code for pre- or post-processing.

API and Request Structure

The Inference Toolkit uses an API similar to the Hugging Face Accelerated Inference API. Requests must define inputs within an inputs key, and optional pipeline parameters can be added via a parameters key.

Example request structures include:

  • Text Classification: A JSON body containing the inputs string.
  • Question Answering: An inputs object containing both question and context keys.
  • Zero-Shot Classification: An inputs string and a parameters object containing candidate_labels.

Customization and "Bring Your Own Code"

While zero-code deployment is the primary feature, the Inference Toolkit supports "bring your own code" methods. This allows developers to override default methods to implement custom inference logic.

Deployment Workflows

There are three primary workflows for deploying Transformer models to Amazon SageMaker:

1. Deploying Directly After Training

When using the Estimator class to train a model in SageMaker, the model can be deployed immediately after the .fit() method. This approach ensures that the SageMaker model container metadata maintains a lineage from the training job to the deployed model.

2. Deploying from Pre-trained Checkpoints

For models already trained and stored as artifacts on Amazon S3, the HuggingFaceModel class can be used by specifying the model_data argument (the S3 path to the model), the IAM role, and the versions of transformers and pytorch used.

3. Deploying from the Hugging Face Model Hub

Users can deploy any of the 10,000+ publicly available models from the Model Hub directly to SageMaker by defining two environment variables in the HuggingFaceModel configuration:

  • HF_MODEL_ID: The specific model ID from huggingface.co/models.
  • HF_TASK: The NLP task associated with the Transformers pipeline (e.g., question-answering).

Technical Specifications and Compatibility

  • Supported Models: The system supports any Transformer model trained in SageMaker (or other compatible platforms), any public model from the Hugging Face Model Hub, and private models hosted in premium Hugging Face accounts.
  • Supported Tasks: The Inference Toolkit and DLCs support any task available via transformers pipelines.
  • Deployment Tools: While the SageMaker Python SDK is the primary interface, the Hugging Face DLCs can be deployed using other tools including the AWS CLI, boto3, Terraform, and CloudFormation templates.
  • Security: Deployments are secured via AWS mechanisms, including encryption at rest and in transit, Virtual Private Cloud (VPC) connectivity, and Identity and Access Management (IAM).

Sources