Deploying Hugging Face ViT on Kubernetes with TF Serving

Hugging Face has detailed a workflow for scaling the deployment of Vision Transformer (ViT) models from the 🤗 Transformers library using Docker and Kubernetes with TensorFlow Serving. This approach allows developers to move from local prototypes to production environments that can handle high user traffic through containerization and automated orchestration.

Scaling Model Deployment with Docker and Kubernetes

To scale a model deployment for real-world projects, Hugging Face recommends a two-step workflow: containerizing the application logic and deploying those containers on a Kubernetes cluster. While managed services like SageMaker or Vertex AI exist, using Docker and Kubernetes provides more granular control over the deployment while abstracting complex infrastructure management.

In this implementation, Google Kubernetes Engine (GKE) is used to provision and manage the cluster, though the concepts are applicable to other platforms like Amazon EKS or local tools like Minikube.

Containerization Process with Docker

TensorFlow Serving requires models to be in the SavedModel format and organized in a specific directory structure: <MODEL_NAME>/<VERSION>/<SavedModel>. This structure allows TensorFlow Serving to manage multiple versions of a model simultaneously.

Building the Custom Image

The process for creating a production-ready image involves:

  1. Directory Setup: Placing the SavedModel into a structured path (e.g., models/hf-vit/1).
  2. Base Image Integration: Running the official tensorflow/serving image and copying the models directory into the running container.
  3. Image Commitment: Using docker commit to create a new image and setting the MODEL_NAME environment variable (e.g., hf-vit) to tell TensorFlow Serving which model to deploy.

Local Testing and Distribution

Before cluster deployment, the image is tested locally by mapping ports 8500 (gRPC) and 8501 (HTTP/REST). Once verified, the image is pushed to a registry, such as Google Container Registry (GCR), using gcloud auth configure-docker and docker push.

Kubernetes Deployment and Orchestration

Deploying to a Kubernetes cluster involves provisioning the infrastructure and defining the desired state of the application through YAML manifests.

Infrastructure Provisioning

Using GKE, a cluster is provisioned with specific machine types (e.g., n1-standard-8) and a set number of nodes. Authentication and connection are handled via the gcloud container clusters get-credentials command.

Kubernetes Manifests

Three primary manifest files are used to separate concerns:

  • deployment.yaml: Defines the Docker image URI, resource limits (e.g., cpu: 800m), and TensorFlow Serving specific arguments. Key tuning parameters include tensorflow_inter_op_parallelism (recommended value: 2) and tensorflow_intra_op_parallelism (recommended value: number of physical CPU cores).
  • service.yaml: Configures a LoadBalancer to expose the gRPC (port 8500) and REST (port 8501) endpoints externally.
  • hpa.yaml: Implements a Horizontal Pod Autoscaler (HPA) to automatically adjust the number of replicas (e.g., between 1 and 3) based on a targetCPUUtilizationPercentage (e.g., 80%).

For managing multiple manifests, Hugging Face suggests using Kustomize, which allows for a single-command deployment via kustomize build . | kubectl apply -f -.

Testing and Optimization

Endpoint Verification

Once the external IP is retrieved via kubectl get svc, the endpoint can be tested by sending base64-encoded image bytes to the REST API. A successful request returns the predicted label and confidence score.

TensorFlow Serving Configurations

To further optimize performance, TensorFlow Serving offers several configurations:

  • enable_batching: Collects incoming requests within a timing window to perform batch inference, which is highly efficient for applications that do not require instant individual predictions.
  • enable_model_warmup: Uses dummy input data to instantiate TensorFlow components lazily, eliminating initial lags during actual service time.

Hardware Optimization

TensorFlow Serving can be accelerated using hardware-specific instruction sets like AVX512. Using an optimized build of the TensorFlow Serving image tailored to the deployment hardware can significantly speed up deep learning model inference.

Sources