mlcommons/inference

What it solves

MLPerf Inference is a standardized benchmark suite designed to measure the performance of AI systems. It solves the problem of inconsistent performance reporting by providing a consistent set of models, datasets, and deployment scenarios (such as edge and datacenter) to evaluate how fast systems can run AI models in real-world settings.

How it works

The suite provides reference implementations for a wide variety of AI models across different modalities—including text (LLMs like Llama 3.1, DeepSeek-R1, BERT), image (ResNet, YOLO, Stable Diffusion XL), video (Wan2.2), audio (Whisper), and multimodal (Qwen3-VL). It allows users to run these models on their hardware and software stacks using various frameworks (PyTorch, TensorFlow, ONNX, TVM) to measure speed and accuracy.

Who it’s for

Hardware and software developers, AI researchers, and system architects who need to objectively compare the performance of AI inference engines, accelerators, and deployment frameworks across different categories of hardware (edge vs. datacenter).

Highlights

• Broad Model Coverage: Supports a massive range of models including LLMs, VLMs, text-to-video, and recommendation systems (DLRM).
• Standardized Scenarios: Differentiates between "edge" and "datacenter" deployment categories to reflect different hardware constraints.
• Flexible Frameworks: While reference implementations are provided, submitters can use their own frameworks to optimize performance.
• Multi-modal Benchmarking: Covers everything from vision, language, and speech-to-text to graph neural networks and RAG (Retrieval-Augmented Generation) systems.