Google Cloud C4 and Intel Xeon 6 Performance for GPT OSS
Google Cloud C4 Virtual Machines (VMs) powered by Intel Xeon 6 processors (Granite Rapids) provide a 1.7x improvement in Total Cost of Ownership (TCO) for GPT OSS Large Language Model (LLM) inference compared to previous-generation C3 instances. This efficiency is driven by a combination of next-generation hardware and targeted software optimizations in the Hugging Face transformers library.
Performance Gains and TCO Improvement
Google Cloud C4 VMs demonstrate superior throughput and cost-efficiency over C3 VMs when running the GPT OSS model. At a batch size of 64, C4 instances provide 1.7x the throughput per vCPU compared to C3 instances. Because the hourly cost scales linearly with vCPU count, this performance increase translates directly into a 1.7x TCO advantage, meaning C3 instances would require 1.7x the expenditure to generate the same volume of tokens.
Key metrics include:
- Throughput: 1.4x to 1.7x TPOT (Time Per Output Token) throughput per vCPU per dollar.
- Cost: Lower price per hour compared to C3 VMs.
Technical Optimizations for MoE Inference
GPT OSS is an open-source Mixture of Experts (MoE) model. MoE architectures use a gating network to route inputs to specialized "expert" sub-networks, allowing the model to scale capacity without a linear increase in compute costs. Because only a small subset of experts is activated per token, CPU inference becomes a viable option for these large models.
To further enhance efficiency, Intel and Hugging Face implemented an expert execution optimization (merged via PR #40304) in the transformers library. This optimization eliminates redundant computation by ensuring each expert only processes the tokens it is specifically routed to, rather than processing all tokens. This change removes wasted FLOPs and improves overall hardware utilization.
Benchmark Methodology and Hardware
Benchmarks were conducted using a controlled, repeatable generation workload to isolate the architectural differences between Intel Xeon 6 (GNR) and 4th Gen Intel Xeon (SPR) processors.
Hardware Configuration
| Instance | Architecture | vCPUs |
|---|---|---|
| C3 | 4th Gen Intel Xeon processor (SPR) | 172 |
| C4 | Intel Xeon 6 processor (GNR) | 144 |
Workload Parameters
- Model:
unsloth/gpt-oss-120b-BF16 - Precision: bfloat16
- Input/Output Length: 1024 tokens each
- Batch Sizes: 1, 2, 4, 8, 16, 32, 64
- Optimizations: Static KV cache and SDPA (Scaled Dot Product Attention) attention backend for determinism.
Conclusion
The collaboration between Intel and Hugging Face demonstrates that large MoE models can be efficiently served on next-generation general-purpose CPUs. By combining the Intel Xeon 6 architecture in Google Cloud C4 VMs with specific framework optimizations, developers can achieve higher throughput, lower latency, and reduced operational costs for large-scale LLM inference.