huggingface/optimum-benchmark

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.

What is Optimum-Benchmark?

Optimum-Benchmark is a tool for measuring how well machine learning models perform on different hardware and software setups. It's built by Hugging Face, the company behind many popular open-source AI libraries.

Think of it as a stopwatch and measuring tape for AI models — it tells you how fast a model runs, how much memory it uses, how much energy it consumes, and how many requests it can handle per second. The key feature is that it works across many different model libraries, hardware types, and optimization techniques, all through one unified interface.

Why does it exist?

The README lists three main motivations:

  1. Hardware comparison: Hardware vendors want to know how their chips perform compared to competitors when running the same models.
  2. Model comparison: Users want to know how their chosen model performs in terms of latency, throughput, memory usage, and energy consumption compared to other models.
  3. Optimization testing: Researchers and engineers want to benchmark different optimization and quantization schemes (techniques that make models smaller or faster) to see which ones actually help.

How does it work?

A benchmark run is defined by three components:

  • Launcher — how the benchmark process is started. Options include process (runs in a single isolated process), torchrun (runs across multiple processes for distributed computing), and inline (for debugging only, not recommended for real benchmarks).
  • Scenario — what you're measuring. Either inference (how fast the model produces outputs) or training (how fast the model learns).
  • Backend — which software library and hardware device to run on. Examples include PyTorch on CPU, ONNXRuntime on CUDA, vLLM on ROCM, etc.

What does it measure?

For both inference and training scenarios, it can track:

  • Latency — how long each operation takes
  • Throughput — how many operations complete per unit of time
  • Memory usage — how much RAM/VRAM the model consumes
  • Energy consumption — how much power the hardware draws

It also supports warm-up runs before measuring, and lets you control input shapes (like sequence length for text models) and pass custom arguments to the model (like how many images a diffusion model should generate per prompt).

What backends and devices are supported?

According to the README, these are fully supported (marked with ✅):

  • PyTorch — CPU and CUDA (NVIDIA GPUs)
  • ONNXRuntime — CPU, CUDA, ROCM (AMD GPUs), and TensorRT execution providers
  • Py-TXI — CPU and GPU (this wraps Hugging Face's TGI and TEI inference servers)
  • Neural Compressor — CPU
  • TensorRT-LLM — CUDA
  • OpenVINO — CPU and GPU
  • vLLM — CUDA, ROCM, and CPU

Additionally, the news section mentions a LlamaCpp backend for benchmarking llama-cpp-python bindings.

One backend is listed as not yet supported: PyTorch on Habana Gaudi Processor (HPU).

How do you install it?

The recommended method uses uv, a fast Python package manager:

uv add optimum-benchmark

Or with traditional pip:

pip install optimum-benchmark

You can also install extra dependencies for specific backends, like optimum-benchmark[openvino] or optimum-benchmark[vllm].

How do you run a benchmark?

There are two ways:

1. Python API — Write a Python script that imports the library, configures a benchmark (choosing launcher, scenario, and backend), and runs it. This works well with uv scripts, which automatically create an isolated environment.

2. Hydra CLI — Use the command line with a YAML configuration file:

optimum-benchmark --config-dir examples/ --config-name cuda_pytorch_bert

This produces several output files in a runs/ directory:

  • benchmark_config.json — the exact configuration used
  • benchmark_report.json — the full results (latency, memory, energy, etc.)
  • benchmark_report.txt and benchmark_report.md — human-readable versions of the report
  • benchmark.json — config and report combined
  • benchmark.log — the run's log output

The CLI also supports overriding configuration values (e.g., changing the model or device) and running sweeps across multiple configurations with the --multirun flag.

Current status

The README explicitly states: "Optimum-Benchmark is a work in progress and is not yet ready for production use." It's actively developed, with over 300 tests run on every pull request across CPU, CUDA, and ROCM environments. The project also hosts the codebase for Hugging Face's LLM-Perf Leaderboard.

Notable recent additions (from the news section)

  • A PyPI package is now available (pip install optimum-benchmark)
  • Model loading latency/memory/energy tracking for all backends
  • numactl support to control NUMA nodes (for fine-grained CPU performance control)
  • Four minimal Docker images (CPU, CUDA, ROCM) for reproducible testing
  • A vLLM backend for benchmarking vLLM's inference engine
  • A Py-TXI backend for benchmarking Py-TXI
  • A Python API for running isolated and distributed benchmarks

How to contribute

The project welcomes contributions. The README specifically mentions wanting: more backends (TensorFlow, TFLite, JAX), more tests for optimizations and quantization schemes, more hardware support (HPU, Apple M-series), and task evaluators for common tasks.

Related

  • Project
  • Dispatch
  • Dispatch
  • Dispatch
  • Dispatch