Hugging Face LLM Inference Container for Amazon SageMaker

Hugging Face has released a new LLM Inference Container (DLC) for Amazon SageMaker, allowing users to deploy open-source Large Language Models (LLMs) in a secure, managed environment. This container is powered by Text Generation Inference (TGI), a purpose-built solution designed to optimize the serving of LLMs for high concurrency and low latency.

High-Performance Inference with Text Generation Inference (TGI)

The Hugging Face LLM DLC leverages TGI to provide several technical optimizations for the most popular open-source LLM architectures. These optimizations include:

  • Tensor Parallelism and Custom CUDA Kernels: Distributes model weights across multiple GPUs to handle larger models and increase speed.
  • Continuous Batching: Increases total throughput by batching incoming requests dynamically.
  • Flash-Attention: Uses optimized transformers code for inference to improve efficiency.
  • Quantization: Supports bitsandbytes for reducing model memory footprints.
  • Accelerated Weight Loading: Utilizes safetensors to reduce start-up times.
  • Advanced Generation Controls: Includes logits warpers (temperature scaling, top-k, repetition penalty), stop sequences, log probabilities, and token streaming via Server-Sent Events (SSE).
  • Watermarking: Implements watermarking for Large Language Models to track generated content.

Supported Model Architectures

The LLM Inference Container officially supports a wide range of model architectures, including:

  • BLOOM / BLOOMZ
  • Llama (including Vicuna, Alpaca, and Koala)
  • Falcon 7B / 40B
  • GPT-NeoX 20B (including Pythia, Lotus, Rosey, Chip, RedPajama, and Open Assistant)
  • StarCoder / SantaCoder
  • FLAN-T5-XXL (T5-11B)
  • Galactica
  • MT0-XXL

Deployment Workflow on Amazon SageMaker

Deploying an LLM using the new DLC involves a streamlined process via the sagemaker Python SDK. The workflow consists of the following steps:

1. Container Retrieval

Users retrieve the specific container URI using the get_huggingface_llm_image_uri method, specifying the backend, region, and version (e.g., version 1.0.3).

2. Model Configuration

Models are deployed using the HuggingFaceModel class. Key configuration parameters include:

  • HF_MODEL_ID: The model ID from the Hugging Face Hub (e.g., OpenAssistant/pythia-12b-sft-v8-7k-steps).
  • SM_NUM_GPUS: The number of GPUs to be used per replica.
  • MAX_INPUT_LENGTH and MAX_TOTAL_TOKENS: Constraints on input and output lengths.
  • HF_MODEL_QUANTIZE: An optional setting to enable bitsandbytes quantization for cost optimization.

3. Endpoint Deployment

Models are deployed to an endpoint using the deploy method. For example, a ml.g5.12xlarge instance (featuring 4 NVIDIA A10G GPUs and 96GB of GPU memory) can be used to shard the model across all available GPUs automatically via TGI.

Inference and Generation Parameters

Once deployed, the endpoint supports a variety of generation parameters to control the output of the LLM:

  • Randomness and Sampling: temperature, top_p, top_k, and do_sample.
  • Length and Repetition: max_new_tokens and repetition_penalty.
  • Constraints: stop sequences to terminate generation.
  • Other Controls: seed for reproducibility, typical_p, and return_full_text to determine if the prompt is included in the output.

This infrastructure enables the creation of scalable AI applications, such as chatbots and virtual assistants, by combining the SageMaker endpoint with front-end interfaces like Gradio.

Sources