Deploying GPT-J 6B on Amazon SageMaker with Hugging Face Transformers

Hugging Face has detailed a method for deploying EleutherAI's GPT-J 6B, an open-source 6 billion parameter language model, for real-time inference using Amazon SageMaker and the Hugging Face Inference Toolkit. This approach solves the critical challenge of long model load times, which can otherwise exceed the 60-second request limit imposed by Amazon SageMaker.

Optimizing Model Load Times for Production

Deploying GPT-J 6B into production is challenging due to its memory footprint and loading speed. The weights of the 6 billion parameter model represent approximately 24GB of memory. Loading the model in float32 requires at least 48GB of CPU RAM (one for initial weights and one for the checkpoint).

To improve accessibility, EleutherAI provides float16 weights. When combined with transformers options to reduce memory footprint, the CPU RAM requirement drops to roughly 12.1GB. However, standard loading methods remain slow:

  • Standard loading: Loading the model on a P3.2xlarge AWS EC2 instance takes approximately 3 minutes and 32 seconds.
  • Disk-stored loading: Storing the model on disk reduces this to 1 minute and 23 seconds.

Because Amazon SageMaker has a 60-second limit for requests to respond, these load times make the model unsuitable for scalable, reliable production workloads.

Accelerating Load Times with torch.save

By using torch.save(model, PATH) and torch.load(PATH) instead of the recommended from_pretrained method, loading times for GPT-J can be reduced from 1 minute and 23 seconds to 7.7 seconds—a speedup of approximately 10.5x.

Critical Requirement: To avoid incompatibility, users must align the PyTorch and Transformers versions used when saving the model with those used when loading it.

Deployment Workflow on Amazon SageMaker

To deploy GPT-J 6B for real-time inference, the following workflow is used:

  1. Model Serialization: Load GPT-J using from_pretrained and save it as a .pt file using torch.save().
  2. Artifact Creation: Create a model.tar.gz archive containing the model weights and necessary files (such as tokenizer.json). This artifact is then uploaded to an S3 bucket.
  3. Endpoint Deployment: Use the HuggingFaceModel class from the Amazon SageMaker Python SDK to deploy the model.

For this deployment, Hugging Face recommends the ml.g4dn.xlarge instance type (NVIDIA T4 GPU), which costs approximately $500/month.

Inference and Usage Best Practices

Once the endpoint is deployed, predictions are run via the predictor.predict method. The Hugging Face Inference toolkit allows for customization of the generation process through the parameters attribute in the request payload.

Generation Strategies

  • Greedy Search: The default request method. After the first request, inference time is approximately 3 seconds.
  • Beam Search: Configured by adding "num_beams": 5 to the parameters. Inference time is approximately 3.3 seconds.
  • Custom Parameterization: Users can adjust max_length and temperature to control output length and randomness. For example, a request with a max_length of 512 tokens takes approximately 38 seconds.

Advanced Few-Shot Prompting

For advanced use cases, such as few-shot predictions, the eos_token_id can be used to stop generation on a specific token (e.g., ###). This allows the model to follow a pattern provided in the prompt and stop generating once the pattern is complete. For these tasks, inference times typically range between 15 and 45 seconds.

Sources