Building Cost-Efficient Enterprise RAG Applications with Intel Gaudi 2 and Intel Xeon

Combining Intel Gaudi 2 AI accelerators and Intel Xeon CPUs provides a cost-efficient hardware foundation for enterprise Retrieval-Augmented Generation (RAG) applications. By leveraging the Open Platform for Enterprise AI (OPEA), developers can deploy RAG pipelines that optimize the total cost of ownership (TCO) without sacrificing significant performance.

Hardware Architecture for RAG

An efficient RAG architecture separates the embedding process and the large language model (LLM) inference to optimize for different compute requirements:

  • Embedding Model: Runs on Intel Granite Rapids CPUs. This architecture is designed for high-core performance-sensitive workloads and supports the AMX-FP16 instruction set, which provides a 2-3x performance increase for mixed AI workloads.
  • LLM Inference: Runs on Intel Gaudi 2 accelerators. Gaudi 2 is purpose-built for deep learning training and inference in data centers and cloud environments.

Software Stack and Integration

Intel utilizes the OPEA (Open Platform for Enterprise AI) framework and LangChain to simplify the deployment of RAG applications. The technical implementation involves several key components:

Vector Database and Embeddings

The system uses the rag-redis template in LangChain, employing the BAAI/bge-base-en-v1.5 embedding model and Redis as the vector database. This allows for the ingestion of domain-specific data (such as financial filings) into a searchable vector store.

LLM Deployment via TGI

LLMs are deployed on Gaudi 2 using the Hugging Face Text Generation Inference (TGI) server. This setup supports popular open-source models including Llama, Mistral, and Intel NeuralChat.

Key optimization options for TGI on Gaudi 2 include:

  • Quantization: Enabling FP8 quantization can yield a 1.8x throughput increase compared to BF16.
  • Scaling: Larger models (e.g., 70B parameters) can be distributed across multiple accelerators using the --sharded true and --num_shard parameters.
  • Safety: Content moderation can be integrated using the Meta Llama Guard model.

Performance and Cost Analysis

Benchmarks comparing the Llama2-70B model with 16 concurrent users on four Intel Gaudi 2 accelerators versus four Nvidia H100 GPUs reveal a significant difference in cost-efficiency:

  • Throughput: The H100-based system achieves 1.13x higher throughput than the Gaudi 2 system.
  • Performance per Dollar: The Gaudi 2 system delivers significantly higher value, with the H100 system providing only 0.44x the performance per dollar of Gaudi 2.

These results indicate that while H100s may offer a slight edge in raw throughput, Intel Gaudi 2 provides a more favorable total cost of ownership for enterprise-scale RAG deployments.

Technical Specifications

Gaudi 2 Configuration

  • Hardware: HLS-Gaudi2 with eight Habana Gaudi2 HL-225H Mezzanine cards, two Intel Xeon Platinum 8380 CPUs, and 1TB System Memory.
  • OS: Ubuntu 22.04.03.

H100 Configuration

  • Hardware: Lambda labs instance gpu_8x_h100_sxm5 with 8xH100 SXM and two Intel Xeon Platinum 8480 CPUs, and 1.8TB system memory.
  • OS: Ubuntu 20.04.6 LTS.

Intel Xeon Configuration

  • Hardware: Pre-production Granite Rapids platform with 2Sx120C @ 1.9GHz and 8800 MCR DIMMs with 1.5TB system memory.
  • OS: CentOS 9.

Sources