OpenAI Dota 2 1v1 Bot Results

OpenAI has demonstrated that self-play reinforcement learning can propel a machine learning system from sub-human performance to superhuman capability in Dota 2 1v1. Within one month, the system evolved from barely matching high-ranked players to defeating top professional players, illustrating that self-play allows training data to improve automatically as the agent's skill increases.

Superhuman Performance and Training Timeline

OpenAI's bot achieved superhuman status by scaling compute and implementing algorithmic improvements, resulting in a linear increase in TrueSkill rating over time. The progression of the bot's capability is marked by the following milestones:

March 1st: Initial classical reinforcement learning results in a simple environment (Drow Ranger kiting an Earthshaker).
May 8th: A 1.5k MMR tester reported improving faster than the bot.
Early June: The bot began beating the 1.5k MMR tester.
June 30th: The bot won most games against a 3k MMR tester.
July 8th: The bot achieved its first win against a 7.5k MMR semi-pro tester.
August 7th: The bot defeated Blitz (6.2k) 3–0, Pajkatt (8.5k) 2–1, and CC&C (8.9k) 3–0.
August 9th: The bot defeated Arteezy (10k pro) 10–0.
August 10th: The bot defeated Sumail (8.3k pro) 6–0. Sumail described the bot as "unbeatable."
August 11th: The bot defeated Dendi (7.3k pro) 2–0.

Technical Implementation and Interfaces

The bot operated under standard tournament rules for 1v1 without AI-specific simplifications. It utilized the following interfaces:

Observations: The bot used the Bot API to access features visible to humans, including information on heroes, creeps, couriers, and nearby terrain. The environment is partially observable.
Actions: The bot performed actions through the Bot API at a frequency comparable to humans, such as moving, attacking, and using items.
Feedback: Incentives were provided based on winning the game and basic metrics, including health and last hits.

To facilitate training, OpenAI whitelisted a few dozen item builds and separately trained the initial creep block using traditional reinforcement learning techniques.

Iterative Improvements during The International

OpenAI used a combination of "coaching" and self-play to rapidly iterate on the agent during The International tournament. Key improvements included:

Item Build Adaptation: After Pajkatt won a match using an early magic wand, OpenAI added that specific item build to the training whitelist.
Strategy Evolution: The bot learned to bait opponents into aggression by intentionally losing health in the first wave, a strategy that was later countered through further self-play.
Mechanic Discovery: The bot discovered a game mechanic where casting abilities outside of the enemy's vision prevents the enemy from gaining a wand charge.

Known Exploits and Limitations

Despite its high performance, the bot can be confused by scenarios it has not encountered during training. During a LAN event at The International, players identified three primary exploit archetypes:

Creep Pulling: Attracting lane creeps between the bot's tier 2 and tier 3 towers, causing the tower to die via attrition.
Orb of Venom + Wind Lace: Using these items to gain a movement speed advantage at level 1 for a quick first blood.
Level 1 Raze: High-skill players (6–7k MMR) were able to kill the bot at level 1 by landing 3–5 razes in rapid succession.

Infrastructure and Future Goals

To run Dota 2 in the cloud, OpenAI developed a shim to stub out OpenGL calls and configured cloud GPU instances to simulate a physical monitor attachment to bypass boot errors.

OpenAI also developed a scripted bot as a baseline. While the scripted bot reached 70 last hits in ten minutes on an empty lane, the reinforcement learning bot reached approximately 97 last hits in the same timeframe (the theoretical maximum is 101).

The project's ultimate goal is solving 5v5 Dota 2. To prepare for this, OpenAI has amassed a dataset of 5.8 million expert-level replays (approximately 45 minutes each with 10 humans) to utilize behavioral cloning.