SemiAnalysisAI/InferenceX

InferenceX™ – Open‑Source Inference Research Platform

What it is – InferenceX (formerly InferenceMAX) is a community‑driven benchmark suite that continuously measures the real‑world inference performance of large language models (LLMs) on the latest hardware and software stacks. It ships a collection of reproducible benchmark harnesses, a live public dashboard, and tooling for running the tests on‑premise or in the cloud.

Key components

Directory	Purpose
InferenceX-e2e/	End‑to‑end inference serving benchmarks (throughput, latency, token‑per‑second, etc.).
CollectiveX/	Networking and collective‑communication benchmarks for multi‑node setups (beta).
OperatorX/	Low‑level operator and kernel performance tests (beta).
power_model/	System‑level power‑consumption modelling (open‑source).
shared/	Shared utilities and common code used by the other suites.
experimental/	Miscellaneous experiments that are not yet part of the stable suite.

Why it matters

• Fast‑moving software stack – LLM inference frameworks (vLLM, SGLang, TensorRT‑LLM, etc.) receive performance upgrades daily. Fixed‑point benchmarks become stale quickly. InferenceX runs the latest releases automatically, giving a near‑real‑time view of progress.
• Hardware‑agnostic – Supports a wide range of GPUs/XPUs (NVIDIA H100/H200, AMD MI300X/MI355X, Intel Xe, TPUv7x, etc.) and even experimental chips, letting users compare across vendors.
• Public dashboard – Results are streamed to https://inferencex.com/ where anyone can explore latency/throughput curves for models such as DeepSeek‑V4, Qwen‑3.8, GLM‑5, Kimi‑K3, MiniMax‑M3, etc.
• Open license – Apache 2.0, so anyone can fork, extend, or embed the benchmarks in their own CI pipelines.

Typical workflow

1. Install the harness – Follow the inferencex-e2e/docs/agentx-standalone.md guide to install the agentx client on a server.
2. Select a preset – Use agentx profile <preset> (e.g., minimax-m3-launch) to run the pre‑configured benchmark for a specific model/hardware combo.
3. Submit results – The tool can push metrics to the central dashboard or store them locally for private analysis.
4. Compare – Browse the live dashboard or query the JSON output to compare against other hardware, software versions, or previous runs.

Getting started

# clone the repo
git clone https://github.com/SemiAnalysisAI/InferenceX.git
cd InferenceX
# install dependencies (example for Ubuntu)
sudo apt-get install -y python3-pip cmake build-essential
pip install -r inferencex-e2e/requirements.txt
# run a quick benchmark on the local GPU
agentx profile demo

For detailed CI/Slurm integration see the PR_REVIEW_CHECKLIST.md and AGENTS.md files.

Community & contribution

• PRs are welcomed; the repo includes a full CONTRIBUTING guide and a checklist for benchmark quality.
• The project is backed by a mix of industry partners (OpenAI, Meta, Microsoft, Oracle, AMD, NVIDIA, etc.) and open‑source foundations (PyTorch, vLLM, SGLang).
• A public list of supporters and quotes is available at https://inferencex.semianalysis.com/quotes.

License – Apache License 2.0 (permissive, commercial‑friendly).

Bottom line – InferenceX provides a reproducible, continuously‑updated way to measure how fast LLMs run on today’s hardware, helping researchers, engineers, and hardware vendors track the true performance impact of software optimizations and new chips.