OpenAI Five Benchmark Results

OpenAI Five won a best-of-three series against a team of 99.95th percentile Dota players—including four professional players: Blitz, Cap, Fogged, Merlini, and MoonMeander. This result demonstrates a significant step toward AI systems capable of managing the high complexity and uncertainty inherent in real-world environments.

Match Results and Performance

OpenAI Five displayed varying levels of success depending on the draft and opponent skill level across three distinct match types:

High-Skill Human Competition

In a best-of-three series against the 99.95th percentile human team, OpenAI Five won the first two games. The first game was won in 21 minutes and 37 seconds, while the second was won in 24 minutes and 53 seconds. The human team won the third game after the audience selected an adversarial hero lineup for the AI, resulting in a loss for OpenAI Five after 35 minutes and 47 seconds.

Audience Volunteer Match

In an initial public match against a team of audience volunteers, OpenAI Five won within the first 14 minutes, significantly faster than the typical 45-minute duration of an evenly matched game.

Adversarial Drafting

During the third game against the high-skill team, the audience selected a "Looney-Tunes" lineup for OpenAI Five consisting of Sven, Axe, Slark, Riki, and Queen of Pain. This adversarial selection led to a OpenAI Five to predict its own win probability at only 2.9% before the game began.

Technical Implementation and Drafting

OpenAI Five utilizes a neural network with a win probability output to evaluate the game state and draft matchups.

Automated Drafting

To handle the system's ability to draft heroes, OpenAI Five uses a tree search to find the optimal draft among 11 million possible team matchups. This process involves creating a fake frame for each matchup and using the network's win probability prediction on the first frame to evaluate the potential outcome.

Model Introspection

The AI's internal predictions often differed from human perception. For example, in Game 1, OpenAI Five predicted a 95% win probability despite human observers believing the matchup was even. In Game 2, it predicted a 76.2% win probability.

Training Methodology and Compute

OpenAI Five was trained using a process of continuous parameter initialization across multiple system revisions rather than training from scratch for each version.

Parameter Surgery

OpenAI developed "surgery" tooling to map old parameters to new network architectures. This allowed them to refine specific behaviors, such as splitting a single action head used for both movement and warding into two separate clones to prevent the AI from dropping wards while moving.

Compute Resources

The estimated compute used to train various Dota systems is as follows:

• 1v1 model: 8 petaflop/s-days
• June 6th model: 11 petaflop/s-days
• August 5th model: 35 petaflop/s-days

Model Planning and Prediction

OpenAI provided insights into the model's planning capabilities by training outputs to predict future game states. This includes predicting the location of a hero (e.g., Sven) six seconds into the future, as well as predicting other quantities such as tower counts and last hits.