Hugging Face Embedding Container for Amazon SageMaker Release

Hugging Face has announced the general availability (GA) of the Hugging Face Embedding Container for Amazon SageMaker. This purpose-built inference container allows AWS customers to efficiently deploy open embedding models to build Generative AI applications, specifically Retrieval-Augmented Generation (RAG) systems.

High-Performance Inference via Text Embedding Inference (TEI)

The Hugging Face Embedding Container is powered by Text Embedding Inference (TEI), a memory-efficient solution designed for the high-performance serving of embedding models. TEI optimizes the extraction process for popular model families, including E5, GTE, Ember, and FlagEmbedding.

Key technical features of the TEI-powered container include:

  • Optimized Inference: Utilizes Flash Attention, Candle, and cuBLASLt for optimized transformers code.
  • Efficient Resource Management: Implements token-based dynamic batching and safetensors weight loading.
  • Rapid Deployment: Features small Docker images and fast boot times with no required model graph compilation step.
  • Production Readiness: Includes Prometheus metrics and distributed tracing via Open Telemetry.

Supported Model Architectures

The container supports a wide range of embedding model architectures, including:

  • BERT/CamemBERT: Examples include BAAI/bge-large-en-v1.5 and Snowflake/snowflake-arctic-embed-m-v1.5.
  • RoBERTa: Example includes sentence-transformers/all-roberta-large-v1.
  • XLM-RoBERTa: Example includes sentence-transformers/paraphrase-xlm-r-multilingual-v1.
  • NomicBert: Example includes jinaai/jina-embeddings-v2-base-en.
  • JinaBert: Example includes nomic-ai/nomic-embed-text-v1.5.

Deployment and Performance Benchmarks

Deployment is managed via the sagemaker Python SDK. Users retrieve the container URI using the get_huggingface_llm_image_uri method, with separate images available for CPU and GPU instances.

Performance varies significantly based on the selected instance type. Based on tests embedding 1 million tokens (3,900 requests of 256 tokens each) with 10 concurrent threads:

GPU Performance (ml.g5.xlarge)

  • Hardware: 1x NVIDIA A10G GPU.
  • Throughput: Approximately 130 requests per second.
  • Latency: 4ms at 10 concurrent requests.
  • Total Processing Time: ~30 seconds for 1 million tokens.

CPU Performance (ml.c6i.2xlarge)

  • Hardware: 4 Intel Ice-Lake vCPUs, 8GB memory.
  • Throughput: Approximately 5 requests per second (including network latency from Europe to us-east-1).
  • Latency: 2s at 10 concurrent requests (as measured by CloudWatch).
  • Total Processing Time: ~841 seconds for 1 million tokens.

Sources