Benchmarking Language Model Performance on 5th Gen Xeon at GCP

Introduction

The post demonstrates that lightweight agentic AI workloads can run efficiently on CPU‑only instances, as shown by benchmarking text embedding and generation on two Xeon‑based Google Cloud VMs.

Creating Instances

N2 Instance Creation

To create an N2 instance with 96 vCPUs (one Ice Lake socket) in Google Cloud, select the N2 machine type n2-standard-96, set the CPU platform to Ice Lake, keep OS and storage defaults, and click CREATE.

C4 Instance Creation

To create a C4 instance with 96 vCPUs (one Emerald Rapids socket) matching the N2 core count, select the C4 machine type c4-standard-96, optionally enable all‑core turbo for stable performance, keep OS and storage as for N2, and click CREATE.

Setting Up the Environment

The environment is prepared by cloning the optimum‑benchmark repository, checking out commit d58bb2582b872c25ab476fece19d4fa78e190673, building the Docker image, installing optimum‑intel with IPEX support, setting OpenMP thread affinity, and logging in to Hugging Face to access the Llama model.

Benchmarking

Text Embedding Benchmark

The benchmark uses the WhereIsAI/UAE-Large-V1 model with sequence length 128 and batch sizes from 1 to 128, run via optimum-benchmark --config-dir examples/ --config-name ipex_bert after updating the YAML to bind both NUMA nodes.

Text Generation Benchmark

The benchmark uses the meta-llama/Llama-3.2-3B model with input length 256, output length 32, and batch sizes from 1 to 64, run via optimum-benchmark --config-dir examples/ --config-name ipex_llama after updating the YAML to bind both NUMA nodes.

Results and Conclusion

Text Embedding Results

The C4 instance delivers approximately 10× to 24× higher throughput than the N2 instance across the tested batch sizes for text embedding.

Text Generation Results

The C4 instance shows approximately 2.3× to 3.6× higher throughput than the N2 instance for text generation; for batch sizes 1 to 16 the throughput improvement is about 13× without significantly hurting latency, allowing concurrent query serving.

Conclusion

The performance gains stem from Intel AMX and memory improvements in the 5th‑gen Xeon (Emerald Rapids) CPU. With the upcoming Granite Rapids (Xeon 6) offering roughly 2× further gains on Llama 3, the authors anticipate being able to run lightweight agentic AI solutions entirely on CPU to avoid host‑accelerator traffic overheads, pending availability of Granite Rapids instances on Google Cloud.