OpenAI Five: Dota 2 Reinforcement Learning

TL;DR

OpenAI Five is a reinforcement learning system that defeated the Dota 2 world champions (Team OG) on April 13, 2019. This achievement demonstrates that self-play reinforcement learning can scale to solve tasks with long time horizons, imperfect information, and complex, continuous state-action spaces.

Technical Challenges of Dota 2

Dota 2 presents several fundamental challenges for AI systems that are distinct from simpler games like Chess or Go. OpenAI Five was designed to solve these specific complexities:

  • Long Time Horizons: Actions taken early in a game can have consequences that occur minutes later, requiring the system to handle long-term planning.
  • Imperfect Information: Unlike perfect-information games, Dota 2 requires the AI to operate under a "fog of war," operating with only partial visibility of the game state.
  • Complex State-Action Spaces: The game features continuous state-action spaces, which means the AI must navigate a complex environment where precise movements and target selection are a successful strategy.

Scaling Reinforcement Learning

OpenAI Five leveraged existing reinforcement learning techniques, but scaled them to an unprecedented level. The system's training process involved:

  • Distributed Training: OpenAI developed a distributed training system and specialized tools for continual training to maintain stability and stability over long periods.
  • Massive Data Throughput: The system was trained on batches of approximately 2 million frames every 2 seconds.
  • Extended Training Duration: The system was trained for a total of 10 months to reach superhuman performance levels.

Implications for General AI

By defeating Team OG, the world champions, OpenAI Five is a proof of concept for the self-play reinforcement learning paradigm. It shows that the AI can discover and strategies that are effective against the same level of opponent, through iterative self-play, can achieve superhuman performance on a difficult, real-world task. These challenges—long horizons, imperfect information, and continuous spaces—are more likely to be representational of the general AI systems of the future.

Sources