CPU Optimized Embeddings with Optimum Intel and fastRAG

Hugging Face and Intel have demonstrated a method to significantly accelerate embedding models on Intel Xeon CPUs using the Optimum Intel library and fastRAG framework. By employing post-training static quantization to int8, these optimizations reduce latency by up to 4.5x and increase throughput by up to 4x compared to baseline bf16 models.

Accelerating Embedding Models for RAG

Embedding models are critical for Retrieval Augmented Generation (RAG) pipelines, serving three primary functions: offline document indexing, real-time query encoding, and the reranking of retrieved documents. While semantic retrieval captures context better than sparse retrieval (like BM25), it is more computationally intensive.

Optimizing these models on CPU backends is essential for:

  • Document Indexing: Increasing throughput to index large collections faster.
  • Query Encoding: Reducing latency for responsive real-time retrieval.
  • Reranking: Enabling rapid processing of candidate sets for time-sensitive applications.

Technical Optimization via Optimum Intel and IPEX

Optimum Intel is an open-source library designed to accelerate Hugging Face pipelines on Intel hardware. It leverages Intel‡ Advanced Vector Extensions 512 (AVX-512), Vector Neural Network Instructions (VNNI), and Intel‡ Advanced Matrix Extensions (AMX) to optimize deep learning workloads.

Quantization Workflow

The optimization process focuses on reducing precision from fp32 to int8 using the following steps:

  1. Installation: Utilizing optimum[neural-compressor] and intel-extension-for-transformers.
  2. Post-training Static Quantization: This process uses a calibration set (such as a subset of the qasper dataset) to determine the dynamic range of weights and activations, minimizing accuracy loss during the transition to int8.
  3. Inference: Quantized models are loaded via IPEXModel for optimized runtime execution using the Intel Extension for PyTorch (IPEX).

Performance Benchmarks for BGE Models

Evaluations were conducted using BGE (Beijing Academy of Artificial Intelligence) models in small (45M), base (110M), and large (355M) parameter sizes on 4th gen Intel Xeon 8480+ CPUs.

Accuracy Retention

Quantization to int8 results in minimal accuracy degradation compared to fp32 models across MTEB (Massive Text Embedding Benchmark) tasks:

  • Reranking: Less than 1% error rate.
  • Retrieval: Less than 1.55% error rate.

Latency and Throughput

Compared to baseline PyTorch bf16 implementations, the int8 quantized models show substantial gains:

  • Latency: Up to 4.5x speedup. Small and base models achieved latency under 10ms, while large models remained under 20ms.
  • Throughput: Up to 4x improvement. Peak throughput for all model sizes was observed at a batch size of 128 for documents of 256 tokens.

Integration with fastRAG

fastRAG is a research framework by Intel Labs for optimized RAG pipelines, compatible with Haystack. The optimized embedding models are integrated into fastRAG through two specific modules:

  • QuantizedBiEncoderRetriever: Used for indexing and retrieving documents from a dense index.
  • QuantizedBiEncoderRanker: Used for reranking retrieved documents to improve relevance.

These modules allow developers to easily swap standard embedding models for Intel-optimized versions within Haystack-based pipelines to improve the efficiency of both the indexing and retrieval phases.

Sources