OpenAI Five defeats Dota 2 world champions
OpenAI Five is the first AI to beat world champions in an esports game, having won two consecutive games against the world champion Dota 2 team, OG. This achievement marks the first time an AI has defeated esports professionals in a live, livestreamed environment.
Scaling Compute as the Primary Driver of Performance
The fundamental requirement for OpenAI Five to reach world-class performance was scale rather than new algorithmic complexity. While the team initially expected to need sophisticated hierarchical reinforcement learning, they found that scaling the training process provided the necessary breakthrough.
To achieve this, OpenAI developed a system called Rapid to run Proximal Policy Optimization (PPO) at an unprecedented scale. The results demonstrated that current reinforcement learning (RL) algorithms are highly powerful when provided with massive amounts of experience, though this remains a challenge for applications outside of simulated environments.
Training Scale and Compute Statistics
The victory over OG was attributed to an 8x increase in training compute compared to the version of OpenAI Five that competed at The International 2018. Because the project had already utilized most available compute for a single model, the team increased scale by extending the training duration.
Key compute metrics for the Finals version of OpenAI Five include:
- Total Compute: 800 petaflop/s-days.
- Simulated Experience: Approximately 45,000 years of Dota self-play over 10 real-time months.
- Daily Experience: An average of 250 years of simulated experience per day.
When evaluated on the current game patch, the Finals version of OpenAI Five maintains a 99.9% win rate against the version used at The International.
Long-term Training and Transfer Learning
OpenAI Five has been training continuously since June 2018. The system demonstrated the ability to transfer learning across changes in model size and game rule updates (patches). This represents a significant milestone in RL, as maintaining a long-lived training run across architecture changes is typically an open challenge.
To facilitate this, OpenAI utilized "surgery" tooling to allow the model to start from previously trained parameters even after substantial architectural modifications.
Hero Pool Expansion and Limitations
The Finals version of OpenAI Five plays with 17 heroes. While the team successfully expanded the pool from 5 to 18 heroes with little training slowdown, attempts to scale to 25 heroes saw a slowdown in learning speed.
Heroes in the 25-hero pool reached approximately 5k MMR (roughly the 95th percentile of players) but did not reach professional levels before the Finals. OpenAI hypothesizes that this may be due to insufficient model capacity, the need for better matchmaking for expanded pools, or the requirement for more training time for new heroes to catch up to established ones.
Zero-Shot Transfer to Human Cooperation
Despite being trained exclusively to compete against other bots, OpenAI Five exhibited a rudimentary ability to act as a teammate for humans. This is an example of zero-shot transfer learning, as the agent generalizes its behavior to cooperate with humans without dedicated cooperative training.
Public Testing via OpenAI Five Arena
To test the exploitability of the agent and the reliability of its performance, OpenAI launched the OpenAI Five Arena. This public experiment allowed the community to interact with the AI in both competitive and cooperative modes, serving as the largest deployment of a highly competent deep reinforcement learning agent for public interaction.