Deploying a Hugging Face Transformers Sentiment Analysis Pipeline on Google Cloud Run

TL;DR

A Hugging Face community post details a step‑by‑step deployment of the distilbert-base-uncased-finetuned-sst-2-english sentiment‑analysis pipeline to Google Cloud Run, showing how to build a lightweight Docker image, configure the service, and achieve sub‑5‑second request latency for under 2,000 monthly requests at a low cost.

Goal of the Micro‑service

The author needed a serverless endpoint that classifies Discord customer reviews as positive or negative. The use case required only a few thousand requests per month, so high throughput and ultra‑low latency were not primary concerns.

The Transformers Library

The Hugging Face transformers library provides a pipeline abstraction that loads a model checkpoint and tokenizer together. A minimal example shows how to obtain a sentiment label:

from transformers import pipeline
classifier = pipeline('sentiment-analysis')
print(classifier('We are very happy to include pipeline into the transformers repository.'))
# [{'label': 'POSITIVE', 'score': 0.9978193640708923}]

The author initially confused the .h5 checkpoint with a Keras SavedModel, but discovered it is a weights file that must be loaded via the pipeline API.

Exploration of Google Cloud Options

The author evaluated four GCP services:

  1. AI‑Platform Prediction – unsuitable because the model is a checkpoint, not a pure TensorFlow SavedModel.
  2. App Engine – encountered TensorFlow system‑dependency errors; PyTorch worked but could only handle two concurrent requests.
  3. Cloud Run – provided the best balance of configurability (memory, vCPU) and simplicity using Docker.
  4. Research – identified the need for custom pre‑ and post‑processing hooks, which are supported by AI‑Platform but not required for the final solution.

Final Serverless Architecture

The production setup consists of four artifacts:

  • main.py – a Flask app that receives a GET request with review and optional api_key, loads the sentiment pipeline, and returns the first result.
  • Dockerfile – builds a Python 3.7 image, installs dependencies, copies the code, exposes port 5000, and runs the app with Gunicorn using a single worker and thread to limit memory usage.
  • requirements.txt – pins Flask 1.1.2, torch 1.7.1, transformers ~4.2.0, and gunicorn ≥20.0.0.
  • Model directory – contains pytorch_model.bin, config.json, and vocab.txt for the DistilBERT SST‑2 model; the rust_model.ot and tf_model.h5 files are unnecessary.

main.py (excerpt)

import os
from flask import Flask, jsonify, request
from transformers import pipeline

app = Flask(__name__)
model_path = "./model"

@app.route('/')
def classify_review():
    review = request.args.get('review')
    api_key = request.args.get('api_key')
    if review is None or api_key != "MyCustomerApiKey":
        return jsonify(code=403, message="bad request")
    classify = pipeline("sentiment-analysis", model=model_path, tokenizer=model_path)
    return classify(review)[0]

if __name__ == '__main__':
    app.run(debug=False, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))

Dockerfile (excerpt)

FROM python:3.7
ENV PYTHONUNBUFFERED True
COPY requirements.txt /
RUN pip install -r requirements.txt
COPY . /app
EXPOSE 5000
ENV PORT 5000
WORKDIR /app
CMD exec gunicorn --bind :$PORT main:app --workers 1 --threads 1 --timeout 0

The single‑worker, single‑thread configuration keeps the memory footprint low (≈4 GB) and avoids spawning multiple instances that would increase billing.

Deployment Steps

  1. Prerequisites – create a GCP project, enable billing, and install the gcloud CLI.
  2. Build the container:
    gcloud builds submit --tag gcr.io/PROJECT-ID/ai-customer-review
  3. Deploy to Cloud Run (managed platform):
    gcloud run deploy --image gcr.io/PROJECT-ID/ai-customer-review --platform managed
  4. Increase memory – after deployment, edit the revision in the Cloud Run console and raise memory from 256 MiB to 4 GiB to accommodate the PyTorch model.

Performance Characteristics

  • Latency – First request (cold start) takes ~10 s; subsequent requests complete in <5 s, including model loading and inference.
  • Optimization tip – Load the pipeline once at module import time (global variable) to eliminate per‑request model loading, reducing both latency and memory churn.

Cost Estimate

Using Google’s pricing calculator, a 4 GiB Cloud Run instance handling ~1,000–2,000 requests per month costs roughly €0.10 per GB of stored container image per month plus negligible compute charges. The Docker image size is ~1 GB (≈700 MB PyTorch + 250 MB model).

Conclusions and Future Work

Deploying a Hugging Face sentiment‑analysis pipeline on Cloud Run provides a quick, cost‑effective solution for low‑volume NLP micro‑services. The approach avoids the need for custom TensorFlow serving, leverages PyTorch for faster model loading, and demonstrates that serverless containers can host transformer models with acceptable latency. Future improvements could include:

  • Providing a pure TensorFlow SavedModel to enable AI‑Platform Prediction.
  • Implementing a warm‑up strategy or background worker to keep the model resident.
  • Exploring a lightweight “lite” version of the model for even lower memory footprints.

Sources