OpenAI Five

OpenAI Five, a team of five neural networks, has started to defeat amateur human teams at Dota 2.

Model Structure

Each hero in OpenAI Five is controlled by a separate single-layer 1024-unit LSTM that processes a 20,000‑dimensional observation and emits actions through eight enumeration‑valued heads.

Training Approach

OpenAI Five learns entirely from self‑play using a scaled‑up Proximal Policy Optimization algorithm running on 256 GPUs and 128,000 CPU cores, collecting about 180 years of gameplay experience per day.

Exploration

The agent explores by playing 80% of games against its current self and 20% against past selves, with randomized lane assignments and unit properties, and is rewarded with net worth, kills, deaths, assists, last hits, etc., minus the opposing team’s average reward.

Coordination

Teamwork emerges from a "team spirit" hyperparameter that anneals from 0 to 1 during training, weighting each hero’s individual reward against the team’s average reward without any explicit communication channel.

System Infrastructure

Training is performed by the Rapid framework, which separates rollout workers gathering experience from optimizer nodes performing synchronous gradient descent, with experience synced via Redis and gradients averaged using NCCL2 wrappers; Rapid runs on Kubernetes, Azure, and GCP backends.

Gameplay Results

OpenAI Five defeated an amateur team with 4.2k MMR (93rd percentile) and, after progressive improvements, beat the best OpenAI employee team (2.5k MMR, 46th percentile), a Valve employee team (2.5‑4k MMR, 46‑90th percentile), and won two of three informal scrims against a semi‑pro team (5.5k MMR, 99th percentile) and an amateur team that trains together.

Differences Versus Humans

OpenAI Five receives the full game state instantly, averages 150‑170 actions per minute (theoretical max 450), and reacts in about 80 ms, faster than human reaction times.

Surprising Findings

Binary win/lose rewards alone slowed learning by an order of magnitude; creep blocking emerged without explicit reward; and the system could defeat strong humans while still containing serious bugs such as rare crashes or large negative rewards at level 25.

Future Plans

The team aims to play a top‑professional lineup at The International in August 2018 under a restricted hero set, will release updates along the way, and will publish a final report after the project completes.