mlcommons/training

MLPerf® Training Reference Implementations (mlcommons/training)

What it is – A collection of reference code for the MLPerf Training benchmark suite. Each benchmark (e.g., Llama 3.1, Stable Diffusion v2, RetinaNet, DLRM, etc.) ships a minimal implementation that can:

• build the model in at least one deep‑learning framework,
• pull the required dataset (via the MLCommons storage site),
• run a Docker container that trains the model until the benchmark’s target quality is reached, and
• report the time taken.

The implementations are deliberately not fully optimized; they are meant as a starting point for anyone who wants to submit a benchmark result or study how the benchmark works.

---

Key points from the README

Aspect	Details
Scope	Reference implementations for many MLPerf training models (vision, language, recommendation, graph, diffusion, etc.).
Components per benchmark	• Model code (usually in PyTorch, TensorFlow, NeMo, or other frameworks)
• Dockerfile
• Dataset‑download script and optional verification script
• Training‑run script that measures time to reach target quality
How to run	1. Install Docker & run the shared install_cuda_docker.sh.
2. Download the dataset from training.mlcommons-storage.org on the host.
3. (Optional) verify the dataset.
4. Build and launch the Docker image using the command provided in each benchmark’s folder.
Benchmarks covered	From early versions (v4.0) up to the current v6.1, including models such as:

• Flux.1 (text‑to‑image, Torchtitan)
• Llama 3.1‑8B, Llama 2‑70B‑LoRA (NeMo / PyTorch)
• GPT‑OSS‑20B, DeepSeek‑V3, Qwen‑35‑397B‑GRPO (large‑scale LLMs)
• RetinaNet, Stable Diffusion v2, BERT‑large, GPT‑3, DLRM, RGAT (vision, NLP, recommendation, GNN) | | Citation | If you use any part of the repo in academic work, cite the MLPerf Training benchmark paper (arXiv:1910.01500) and the provided BibTeX entry. | | Status | Described as “alpha/beta” quality; contributors are encouraged to file issues/PRs to improve them. |

---

Who might find this useful?

• Researchers preparing a submission to the MLPerf Training benchmark and needing a baseline implementation.
• Hardware vendors who want a reproducible workload to test new accelerators (knowing the code is not heavily tuned).
• Students or engineers learning how large‑scale training benchmarks are structured (Docker‑based, dataset handling, quality‑target stopping criteria).

---

Getting started (quick steps)

1. Clone the repo: git clone https://github.com/mlcommons/training.git.
2. Choose a benchmark folder (e.g., text_to_image for Flux.1).
3. Follow the folder’s README: run install_cuda_docker.sh, download the dataset as instructed, then docker build and docker run the provided command.
4. The container will train the model until the benchmark‑defined quality metric is met and then print the elapsed time.

---

Limitations / cautions

• Performance not representative – These are reference (unoptimized) implementations; real‑world performance numbers will be much slower than what a production‑grade stack could achieve.
• Heavy resource requirements – Some benchmarks (e.g., 70B‑parameter LLMs) need large GPU clusters and massive storage; they are not runnable on a typical laptop.
• Dataset download outside Docker – The scripts assume the dataset resides on the host filesystem; mis‑placement can cause failures.
• Alpha/Beta quality – Code may be incomplete, contain bugs, or lack extensive testing. Community contributions are expected to improve stability.

---

Bottom line

mlcommons/training is a genuine, community‑maintained repository that provides minimal, reproducible reference implementations for the MLPerf Training benchmark suite across a wide range of AI tasks. It is a valuable starting point for anyone looking to understand, experiment with, or submit results to the MLPerf training challenges, but it is not intended for production‑level training performance.