Deploying TensorFlow Vision Models in Hugging Face with TF Serving

Hugging Face has detailed a workflow for deploying TensorFlow-based vision models from the Transformers library using TensorFlow Serving (TF Serving). This approach allows developers to expose state-of-the-art models—including Vision Transformer (ViT), Masked Autoencoders, RegNet, and ConvNeXt—as high-performance REST or gRPC endpoints with built-in support for server-side batching and model warmup.

Saving Models for TF Serving

To deploy a model with TF Serving, the model must be in the SavedModel format. TensorFlow models in the Hugging Face Transformers library include a save_pretrained() method that can serialize weights into both h5 and SavedModel formats.

For a Vision Transformer (ViT) model, the process involves loading the model via TFViTForImageClassification.from_pretrained() and calling save_pretrained(saved_model=True). By default, this creates a versioned directory structure (e.g., {model_dir}/saved_model/{version}) required by TF Serving.

Implementing Model Surgery for Pre- and Post-processing

Standard ML models require specific preprocessing and post-processing. To reduce training-serving skew and lower the cognitive load for developers, these operations can be embedded directly into the model's computation graph through "model surgery."

Preprocessing Pipeline

For ViT models, essential preprocessing steps include:

  • Scaling: Converting image pixel values to a [0, 1] range.
  • Normalization: Scaling pixel values to a [-1, 1] range using the model's specific mean and standard deviation.
  • Resizing: Adjusting the image to a spatial resolution of 224x224.
  • Transposition: Moving the channel dimension to the front to match the channel-first format used by Hugging Face models.

To optimize request payloads and prevent size inflation, the guide recommends accepting base64-encoded strings as input, which are then decoded and processed within the graph using tf.io.decode_base64 and tf.io.decode_jpeg.

Post-processing and Export

Post-processing transforms raw model logits into human-readable labels. By deriving a concrete function from the model's call() method, developers can wrap the model in a serving_fn that:

  1. Executes the preprocessing pipeline.
  2. Runs the model inference.
  3. Applies a softmax function to the logits to determine confidence scores.
  4. Maps the resulting indices to string labels using the model's id2label configuration.

This wrapped function is then exported as the serving_default signature using tf.saved_model.save(), changing the model's input requirement from a 4D tensor to a string and its output to a dictionary containing the label and confidence score.

Deployment with TensorFlow Serving

Once the model is exported, it can be deployed using the tensorflow_model_server command. Key configuration parameters include:

  • rest_api_port: The port for the REST endpoint (default is often 8501).
  • model_name: The identifier used when calling the API.
  • model_base_path: The directory where TF Serving loads the latest model version.

TF Serving provides two primary ways to query the deployed model:

REST Endpoint

The REST API is suitable for online prediction scenarios. Requests are sent as JSON payloads containing base64-encoded images in an instances list. The endpoint follows the format: http://localhost:8501/v1/models/{model_name}:predict.

gRPC Endpoint

For low-latency, highly scalable, and distributed systems, gRPC is the preferred deployment method. This requires opening a communication channel via grpc.insecure_channel and using a PredictionServiceStub to send PredictRequest payloads. gRPC returns structured outputs containing the confidence score and the predicted label.

Sources