Deploying Hugging Face ViT on Vertex AI

Hugging Face has detailed a workflow for deploying Vision Transformer (ViT) models on the Vertex AI platform. This approach allows developers to achieve the scalability of a Kubernetes-based deployment while reducing the amount of code required for infrastructure management.

Vertex AI Capabilities for Model Deployment

Vertex AI is a managed machine learning platform that provides a unified API for the entire ML workflow. For model deployment, it offers several production-ready features:

  • Autoscaling: Automatically adjusts resources based on incoming traffic.
  • Traffic Management: Supports traffic splitting between different model versions and model versioning for easy rollbacks.
  • Operational Control: Includes authentication, rate limiting, and integrated model monitoring and logging.
  • Prediction Support: Handles both online and batch prediction requests.

While the guide focuses on TensorFlow, Vertex AI also supports PyTorch and scikit-learn frameworks.

The Serving Model Architecture

The deployment utilizes a ViT B/16 model implemented in TensorFlow. To minimize training-serving skew, the model is serialized as a SavedModel with pre-processing and post-processing operations embedded directly into the graph.

Model Input and Output:

  • Input: The model accepts base64 encoded strings of images to prevent modification during network transmission. Pre-processing steps include resizing images to 224x224, standardizing them to a [-1, 1] range, and transposing them to a channels_first memory layout.
  • Output: The model returns the predicted label (as a string) and the confidence score (as a float).

Model artifacts must be stored in a Google Cloud Storage (GCS) bucket before deployment to Vertex AI.

Deployment Workflow

Deployment is managed via the google-cloud-aiplatform Python SDK and follows a four-step process:

1. Model Upload

The SavedModel is uploaded to the Vertex AI Model Registry, a fully managed registry that handles storage and versioning. This step requires a container_spec pointing to a pre-built Docker image provided by Vertex AI for TensorFlow serving.

2. Endpoint Creation

An Endpoint is created to serve as the interface for receiving requests and sending responses. This provides a managed entry point for the model.

3. Model Deployment

The uploaded model is linked to the endpoint. During this phase, hardware specifications are defined under dedicated_resources:

  • Machine Type: Defines the base compute (e.g., n1-standard-8 with 8 vCPUs and 32GB RAM).
  • Accelerator: Specifies the GPU type (e.g., NVIDIA_TESLA_T4) and the number of accelerators per replica.
  • Scaling: Sets the min_replica_count and max_replica_count for autoscaling.

4. Prediction Requests

Predictions are made using the PredictionServiceClient. For binary data like images, the request payload must be formatted as: {serving_input: {"b64": base64_encoded_string}}.

Performance and Monitoring

Vertex AI provides built-in monitoring tools to track endpoint performance and resource utilization, such as the accelerator duty cycle.

Load Testing Results: Using Locust for local load testing, the team processed approximately 17,230 requests with an average latency of 646 milliseconds.

Pricing Overview

Costs are based on the hourly rate of the compute node and any attached accelerators. Based on the examples used in the post:

Resource Specification Hourly Pricing (USD)
Machine Type n1-standard-8 (8vCPU, 30GB) $0.4372
Accelerator NVIDIA_TESLA_T4 $0.4024

Users are charged only when the node is processing actual prediction requests.

Sources