Deploying a Serverless Transformers Pipeline on Google Cloud

A Hugging Face community member, Maxence Dominici, has detailed a workflow for deploying a sentiment analysis microservice on Google Cloud Platform (GCP). The final implementation uses a serverless architecture via Google Cloud Run, allowing for a cost-effective deployment of the distilbert-base-uncased-finetuned-sst-2-english model for low-volume request patterns.

Serverless Deployment via Google Cloud Run

Google Cloud Run is the selected production environment because it allows for flexible memory and vCPU configuration, which is necessary for loading transformer models. The final architecture consists of a Flask application wrapped in a Docker container, deployed as a managed service.

Technical Components

  • Model: The distilbert-base-uncased-finetuned-sst-2-english model is used, specifically the PyTorch version (pytorch_model.bin, config.json, and vocab.txt).
  • Application Logic: A main.py file handles GET requests, requiring a review string and an API key for basic security.
  • Containerization: A Dockerfile based on python:3.7 uses gunicorn as the entrypoint to serve the Flask app.
  • Dependencies: The environment requires Flask==1.1.2, torch===1.7.1, transformers~=4.2.0, and gunicorn>=20.0.0.

Resource Configuration

To avoid memory errors and control costs, the instance is configured with:

  • Memory: Upgraded from the default 256 MB to 4 GB.
  • Concurrency: The Gunicorn configuration is set to --workers 1 --threads 1. This ensures only one process and one thread are active, preventing multiple instances from consuming excessive memory and increasing billing.

Evaluation of GCP Services

Before settling on Cloud Run, several Google Cloud services were tested to determine the most viable path for deploying a Hugging Face pipeline:

Service Outcome Reason for Rejection
AI-Platform Prediction Failed The model is a checkpoint, not a "pure TensorFlow" saved model; beta stability issues were encountered.
App Engine Failed Encountered missing system dependency files during TensorFlow installation; PyTorch worked but could not handle more than two requests per instance.
Cloud Run Success Provided the necessary memory and vCPU controls via Docker images.

Performance and Cost Analysis

Latency

Request handling typically takes less than five seconds, including model loading and prediction. Cold starts can add approximately 10 seconds of additional latency. The author notes that performance could be further improved by "warming

Sources