Deploying Embedding Models with Hugging Face Inference Endpoints

Hugging Face has integrated Text Embeddings Inference (TEI) into its Inference Endpoints service, allowing developers to deploy open-source embedding models with high throughput and significantly lower costs than proprietary alternatives. This integration is specifically designed to optimize Retrieval Augmented Generation (RAG) workflows, where fast and efficient vector representations of data are critical for quality generation.

Text Embeddings Inference (TEI) Capabilities

Text Embeddings Inference (TEI) is a purpose-built solution for serving open-source text embedding models. It is optimized for high-performance extraction and supports the top 10 models from the Massive Text Embedding Benchmark (MTEB) Leaderboard, including E5, GTE, Ember, and FlagEmbedding.

To achieve industry-leading throughput and cost efficiency, TEI implements several technical optimizations:

  • Inference Optimization: Utilizes Flash Attention, cuBLASLt, and the Candle framework for optimized transformers code.
  • Efficient Loading: Employs Safetensors for weight loading and eliminates the model graph compilation step.
  • Operational Efficiency: Features small Docker images for fast boot times (supporting serverless patterns) and token-based dynamic batching.
  • Production Readiness: Includes Prometheus metrics and distributed tracing via Open Telemetry.

Performance and Cost Benchmarks

Deploying open-source models via TEI can result in substantial cost reductions compared to closed-source APIs. In a benchmark using the BAAI/bge-base-en-v1.5 model on an Nvidia A10G Inference Endpoint with a batch size of 32 and a sequence length of 512 tokens, Hugging Face achieved:

  • Throughput: 450+ requests per second.
  • Cost: $0.00000156 per 1k tokens ($0.00156 per 1M tokens).

According to Hugging Face, this represents a 64x cost saving compared to OpenAI Embeddings, which are priced at $0.0001 per 1k tokens.

Deploying via Hugging Face Inference Endpoints

Inference Endpoints provide a managed SaaS environment that removes the need for manual infrastructure management or MLOps. Key features include:

  • Simplified Deployment: Models can be deployed as production-ready APIs in a few clicks.
  • Scaling and Cost Control: Supports automatic scale-to-zero to reduce costs during periods of inactivity.
  • Enterprise Security: Offers SOC2 Type 2 certification, GDPR data processing agreements, BAA, and secure offline endpoints via direct VPC connections.
  • Broad Support: Out-of-the-box compatibility with Sentence-Transformers, Diffusers, and 🤗 Transformers.

Deployment Workflow

To deploy an embedding model, users select a model repository (such as BAAI/bge-base-en-v1.5), choose a cloud provider and region, and select an instance type. While the system suggests instance types based on model size, high-performance benchmarks are achieved using GPU instances like the Nvidia A10G. The deployment process typically takes between 1 to 3 minutes.

Using the Embedding Endpoint

Once an endpoint is online, it can be accessed via an Inference Widget for manual testing or through API requests using cURL, Python, or JavaScript.

Batch Processing and Truncation

TEI supports batch requests, allowing multiple documents to be sent in a single request to increase endpoint utilization. Users should note that TEI does not automatically truncate input; truncation must be explicitly enabled by setting truncate: true in the request payload.

Example Python request for batch embeddings:

import requests

API_URL = "https://your-endpoint-url.huggingface.cloud"
headers = {
    "Authorization": "Bearer YOUR TOKEN",
    "Content-Type": "application/json"
}

output = requests.post(API_URL, headers=headers, json={
    "inputs": ["sentence 1", "sentence 2", "sentence 3"],
    "truncate": True
}).json()

Sources