huggingface/optimum-benchmark

What is Optimum‑Benchmark?

Optimum‑Benchmark is a Python library that lets you measure how fast and how efficiently Hugging Face models run on many different hardware back‑ends. It supports the major model families (Transformers, Diffusers, PEFT, TIMM, etc.) and a wide range of runtimes – PyTorch, ONNX Runtime, OpenVINO, vLLM, TensorRT‑LLM, IPEX, Llama‑Cpp, Py‑TXI and more – on CPUs, GPUs, AMD ROCm, Intel XPU, Habana Gaudi, etc.

The tool can benchmark inference (latency, throughput, memory, energy) as well as training (same metrics plus dataset‑shape control). It works in single‑process mode, with torchrun for distributed runs, or via a simple Hydra‑based CLI. Results are saved as JSON, Markdown and plain‑text files and can be pushed to the Hugging Face Hub for sharing.

---

Who might use it?

• Hardware vendors who want to compare their chips against competitors on identical models.
• Model developers / researchers who need to know the latency, memory footprint or energy cost of a model on a specific backend before deployment.
• ML engineers looking to evaluate the impact of quantisation, pruning or other optimisations provided by the Optimum ecosystem.
• Benchmarking enthusiasts who want reproducible, configurable runs and the ability to sweep over devices, back‑ends or model variants.

---

Core concepts

Concept	What it means
Launcher	How the benchmark process is started – process (isolated), torchrun (distributed) or inline (debug only).
Scenario	What is being measured – inference (forward / generate) or training (trainer loop).
Backend	The runtime that actually executes the model – e.g. pytorch, onnxruntime, vllm, openvino, tensorrt‑llm, etc.
Device	Physical hardware target – cpu, cuda, rocm, gpu, xpu, hpu, etc.
Config	A Hydra‑compatible YAML file (or Python objects) that ties the three pieces together and lets you tweak things like batch size, input shape, warm‑up runs, energy tracking, etc.

---

How to get started

1. Install – the library is on PyPI. The simplest way is pip install optimum-benchmark or, if you prefer the fast uv manager, uv add optimum-benchmark.
2. Pick a backend – install optional extras for the runtime you need, e.g. pip install optimum-benchmark[onnxruntime] or uv add optimum-benchmark --extra vllm.
3. Run a benchmark – either:
   • Python API – create a BenchmarkConfig with a TorchrunConfig, an InferenceConfig (or TrainingConfig) and a backend config (e.g. PyTorchConfig(model="gpt2", device="cuda")). Call Benchmark.launch(config) and inspect the returned BenchmarkReport.
   • CLI – optimum-benchmark --config-dir examples/ --config-name cuda_pytorch_bert. Use --multirun to sweep over values (e.g. backend.device=cpu,cuda).
4. Inspect results – the run writes several files (benchmark_report.json, .md, .txt, etc.) and can optionally push them to the Hugging Face Hub.

---

What makes it useful?

• Unified interface across dozens of back‑ends and devices, so you don’t need separate scripts for each runtime.
• Accurate metrics – latency, throughput, memory usage and optional energy tracking (via codecarbon integration).
• Reproducibility – Hydra configs, Docker images (cpu, cuda, rocm) and CI tests ensure the same benchmark can be rerun anywhere.
• Extensible – you can add new back‑ends, launchers or custom metrics by extending the provided config classes.

---

Where to learn more?

• The README (shown above) contains quick‑start commands and a full list of supported back‑ends.
• Example configuration files live in the examples/ directory of the repo.
• The library’s API docs are generated from the optimum_benchmark package and can be inspected after installation.
• For community‑driven performance comparisons, see the LLM‑Perf Leaderboard hosted on the Hugging Face Hub.

---

TL;DR

Optimum‑Benchmark is a Hugging Face‑maintained, multi‑backend benchmarking suite for transformer‑style models. Install it, pick a backend/device, define a Hydra config (or use the Python API), run the benchmark, and you’ll get detailed latency/memory/energy reports that can be shared publicly. It’s aimed at anyone who needs reliable, reproducible performance numbers across the rapidly evolving AI hardware landscape.