OpenAI Gym Retro Release

OpenAI has released the full version of Gym Retro, a platform designed for reinforcement learning (RL) research on video games. By expanding the available game count from approximately 100 to over 1,000 titles, OpenAI aims to shift RL research from optimizing agents for single tasks to studying how agents generalize across different games with similar concepts but different visual appearances.

Expanded Game Library and Console Support

The full release of Gym Retro significantly increases the scale of the dataset, supporting over 1,000 games across several backing emulators. The platform includes games from the following consoles:

  • Sega Genesis
  • Sega Master System
  • Nintendo NES
  • Nintendo SNES
  • Nintendo Game Boy

Preliminary support has also been added for the Sega Game Gear, Nintendo Game Boy Color, Nintendo Game Boy Advance, and NEC TurboGrafx.

Game Integration Tool

Alongside the platform, OpenAI has released an integration tool that allows researchers to add new games to Gym Retro. Provided the user has the game ROM, the tool enables the following capabilities:

  • Memory Location Discovery: Finding specific memory locations to track game state.
  • Save State Creation: Easily creating save states for agents.
  • Scenario Design: Designing specific scenarios for RL agents to solve.
  • Input Recording: Recording and playing back movie files that store button inputs. These files are compact because they only store the starting state and the sequence of presses rather than full video frames, making them useful for visualizing agent behavior or storing human training data.

Challenges in Reinforcement Learning Generalization

OpenAI identifies two primary levels of generalization difficulty in RL research:

  1. Intra-game Generalization: Generalizing between different levels of the same game (e.g., Sonic The Hedgehog).
  2. Inter-game Generalization: Generalizing between different games entirely, which is a harder problem enabled by the scale of the Gym Retro dataset.

Reward Farming and Faulty Reward Functions

OpenAI observed that agents often engage in "reward farming"—where an agent finds a loop to rapidly accrue points (the defined reward) without completing the actual mission of the game. Examples cited include Cheese Cat-Astrophe and Blades of Vengeance, where characters become trapped in infinite loops to maximize score.

Algorithmic Performance and Reward Density

The effectiveness of current RL algorithms, such as PPO, varies based on the nature of the game's reward structure:

  • Dense Rewards: In games like Gradius, where points are awarded frequently for shooting enemies, RL algorithms perform well because they receive constant feedback and can react frame-by-frame.
  • Sparse Rewards: In games that require long-term planning or provide infrequent rewards, existing algorithms struggle. OpenAI notes that many games in the Gym Retro dataset possess sparse rewards, suggesting that new techniques will be required to solve them.

Sources