huggingface/text-embeddings-inference

What it solves

Text Embeddings Inference (TEI) provides a high-performance solution for deploying and serving open-source text embeddings and sequence classification models. It eliminates the need for complex model graph compilation and reduces boot times, making it suitable for serverless environments.

How it works

TEI acts as a toolkit that serves models via a REST API or gRPC. It utilizes optimized transformers code and backends like Flash Attention, Candle, and cuBLASLt to maximize inference speed. The system supports token-based dynamic batching and weight loading via Safetensors and ONNX to ensure efficient memory and compute usage.

Who it’s for

It is designed for developers and ML engineers who need to deploy embedding models (such as BGE, GTE, and E5) or re-rankers in production environments with low latency and high throughput.

Highlights

• Blazing Fast Inference: Optimized for various NVIDIA GPU architectures (from Turing to Blackwell) and Apple Silicon (Metal).
• Production Ready: Includes distributed tracing with Open Telemetry and Prometheus metrics for monitoring.
• Hugging Face Integration: Direct support for a wide range of model IDs from the Hugging Face Hub.
• Flexible Deployment: Offers small Docker images for fast boot times and supports air-gapped deployments.
• Advanced Batching: Implements token-based dynamic batching to maximize hardware utilization.