huggingface/text-embeddings-inference

What it solves

Text Embeddings Inference (TEI) provides a high-performance solution for deploying and serving open-source text embedding and sequence classification models. It eliminates the need for complex model graph compilation and optimizes the process of extracting embeddings from popular models, making it suitable for production environments and serverless architectures.

How it works

TEI acts as a toolkit that serves models via a REST API or gRPC. It utilizes optimized transformer code and specialized backends to maximize hardware efficiency. Key technical optimizations include:

• Hardware Acceleration: Support for NVIDIA GPUs (across multiple architectures from Turing to Blackwell), Apple Silicon (via Metal), and CPUs.
• Performance Kernels: Integration of Flash Attention, Candle, and cuBLASLt for faster inference.
• Efficient Loading: Uses Safetensors and ONNX for weight loading to improve boot times.
• Request Management: Implements token-based dynamic batching to optimize throughput and handle concurrent requests efficiently.

Who it’s for

It is designed for developers and ML engineers who need to deploy embedding models (like BGE, GTE, or E5) or re-rankers into production with low latency and high throughput.

Highlights

• Blazing Fast: Optimized for high-performance extraction with minimal latency.
• Production Ready: Includes built-in Prometheus metrics and distributed tracing via Open Telemetry.
• Broad Model Support: Compatible with a wide range of architectures including BERT, RoBERTa, Mistral, Qwen, and Gemma.
• Serverless Friendly: Small Docker images and fast boot times facilitate rapid scaling.