AMD EPYC Turin CPU delivers 2× LLM inference throughput over Genoa

TL;DR

AMD’s 5th‑generation EPYC CPU, code‑named Turin, delivers roughly double the inference throughput of the previous‑generation Genoa chips for large language model workloads, thanks to up to 192 cores, 384 threads, and the ZenDNN‑accelerated torch.compile path.

What is AMD Turin?

AMD announced its fifth‑generation server‑grade EPYC processor built on the Zen 5 architecture, marketed as Turin. The chip scales to 192 physical cores and 384 hardware threads, providing a substantial increase in parallel compute capacity over the prior 96‑core Genoa line.

Why Turin matters for Hugging Face users

  • Lower latency – More cores and the ZenDNN‑optimized PyTorch plugin (zentorch) reduce the time to generate each token.
  • Higher throughput – Parallel execution of multiple model instances across all cores enables up to a 2× increase in decoded tokens per second.
  • Cost efficiency – Faster inference translates into fewer servers needed for a given workload, lowering operational expenses.

Benchmark methodology

The Hugging Face team validated the EPYC Turin platform using the ZenDNN 5.0 plug‑in for PyTorch, which integrates with torch.compile to apply graph‑level optimizations. Benchmarks were run in bfloat16 precision on a multi‑instance setup that allocated 32 physical cores per socket to each instance of the Meta LLaMA 3.1 8B‑Instruct model. Two batch sizes (16 and 32) were tested across five representative LLM tasks:

  • Summarization (1024 → 128 tokens)
  • Chatbot (128 → 128 tokens)
  • Translation (1024 → 1024 tokens)
  • Essay writing (128 → 1024 tokens)
  • Live captioning (16 → 16 tokens)

Throughput was measured as decoded tokens per second, excluding the first token to focus on steady‑state performance.

Results: Turin vs. Genoa

Throughput comparison

The chart shows that Turin consistently outperforms Genoa, achieving approximately 2× higher throughput across most batch sizes and use cases. This improvement is attributed to the higher core count and the efficiency gains from the ZenDNN‑accelerated torch.compile pipeline.

Reproducibility tools

  • optimum‑benchmark – Hugging Face’s unified benchmarking framework was used to orchestrate the tests with the zentorch backend.
  • Dockerfile (forthcoming) – An optimized Docker image containing the benchmark code and dependencies will be released, simplifying replication of the results.

Getting started with ZenDNN

Developers can experiment with the AMD Zen Deep Neural Network (ZenDNN) ecosystem through the following open‑source plugins:

  • ZenTF – TensorFlow integration: https://github.com/amd/ZenDNN-tensorflow-plugin
  • ZenTorch – PyTorch integration (used in the benchmarks): https://github.com/amd/ZenDNN-pytorch-plugin
  • ZenDNN ONNXRuntime – ONNX Runtime support: https://github.com/amd/ZenDNN-onnxruntime

For more details, visit the official AMD ZenDNN page: https://www.amd.com/en/developer/zendnn.html.

Conclusion

AMD’s EPYC Turin CPU delivers a significant performance uplift for LLM inference compared with the Genoa generation, enabling Hugging Face users to achieve lower latency, higher throughput, and reduced cost. The combination of ZenDNN‑accelerated torch.compile and the upcoming Docker deployment package makes it straightforward for the community to adopt and benchmark this new hardware generation.

Sources