Hugging Face Transformers v4.2.0 TensorFlow Performance and Serving Updates

Hugging Face has updated its TensorFlow implementations for several key models to increase robustness and inference speed. These improvements specifically target BERT, RoBERTa, ELECTRA, and MPNet, providing performance gains across graph/eager mode, TensorFlow Serving, and CPU, GPU, and TPU devices.

Computational Performance Gains

Transformers v4.2.0 delivers a significant increase in computational speed for TensorFlow models. In benchmarks comparing the v4.2.0 BERT implementation against the official Google implementation on a GPU V100 with a sequence length of 128, the v4.2.0 version is up to ~10% faster.

Batch size Google implementation (ms) v4.2.0 implementation (ms) Relative difference
1 6.7 6.26 6.79%
2 9.4 8.68 7.96%
4 14.4 13.1 9.45%
8 24 21.5 10.99%
16 46.6 42.3 9.67%
32 83.9 80.4 4.26%
64 171.5 156 9.47%
128 338.5 309 9.11%

Additionally, the v4.2.0 implementation is twice as fast as the versions found in the 4.1.1 release.

TensorFlow Serving Integration

To leverage these performance gains in production, Hugging Face has optimized the integration with TensorFlow Serving, a component of TensorFlow Extended (TFX) that allows models to be deployed via HTTP and gRPC APIs.

The SavedModel Format

TensorFlow Serving requires models to be in the SavedModel format, which contains the model architecture and weights in a standalone package. This format allows models to be run without the original source code and supports backends such as C++, Java, Go, and JavaScript.

Since Transformers v4.2.0, creating a SavedModel includes three key enhancements:

  1. Flexible Sequence Length: Sequence length can now be modified freely between runs.
  2. Input Availability: All model inputs are available for inference.
  3. Grouped Outputs: hidden states or attention are grouped into a single output when output_hidden_states=True or output_attentions=True is used.

Custom Serving Signatures

Users can subclass models to define custom serving signatures. This is necessary when passing inputs_embeds (token embeddings) instead of input_ids (token IDs). By overriding the serving method and using the @tf.function decorator with a specific input_signature, developers can define the exact name, data type, and shape of expected inputs for the SavedModel.

Deployment Workflow

Deploying a BERT model for sentiment classification involves a three-step process:

  1. Model Creation: Load a model (e.g., nateraw/bert-base-uncased-imdb) and save it using model.save_pretrained("my_model", saved_model=True) to generate the SavedModel version alongside h5 weights.
  2. Containerization: Use Docker to pull the tensorflow/serving image, copy the SavedModel into the container's models folder, and commit the image with the MODEL_NAME environment variable set.
  3. Inference: Query the deployed model using either the REST API (via HTTP POST requests) or the gRPC API (using tensorflow_serving.apis).

Future Directions

Hugging Face intends to integrate the preprocessing stage directly into the SavedModel to further simplify the deployment pipeline.

Sources