OpenAI Retro Contest and Gym Retro Release

OpenAI has launched the Retro Contest and released Gym Retro to shift reinforcement learning (RL) research away from memorization and toward generalization. The contest evaluates an algorithm's ability to transfer knowledge from training levels to previously unseen video game levels.

The Retro Contest: Testing Generalization in RL

The Retro Contest measures whether an RL algorithm can generalize from previous experience rather than simply memorizing a specific environment. In traditional RL research, algorithms are often tested in the same environment where they were trained, which favors models with many hyperparameters that excel at memorization.

Contest Structure and Constraints

  • Task: Agents are provided with a training set of levels from the Sonic The Hedgehog series and are evaluated on a test set of custom levels created specifically for the contest.
  • Timeline: The contest runs from April 5 to June 5.
  • Training Budget: While any environment or dataset can be used during training, agents are limited to approximately 18 hours (1 million timesteps) on each unseen test level.

Performance Benchmarks

Baseline results indicate that RL algorithms perform significantly worse than humans. Humans playing for only one hour achieved scores far higher than RL algorithms that were given 18 hours of play on test levels, even when those algorithms utilized transfer learning.

OpenAI released a technical report, "Gotta Learn Fast: A New Benchmark for Generalization in RL," which compares several algorithms:

  • PPO (Proximal Policy Optimization): Performance nearly doubled when the network was pre-trained on training levels and then fine-tuned on test levels, demonstrating a reliable effect of transfer learning.
  • Rainbow DQN: Included as a baseline.
  • JERK: A random guessing algorithm optimized for Sonic that replays top-scoring action sequences more frequently as training progresses.

Gym Retro: A New RL Environment Platform

Gym Retro is a system designed to wrap classic video games as RL environments, expanding the complexity and variety of games available for research beyond the Atari 2600.

Technical Capabilities and Scope

  • Game Library: The initial release includes 30 SEGA Genesis games (from the SEGA Mega Drive and Genesis Classics Steam Bundle) and 62 Atari 2600 games from the Arcade Learning Environment.
  • Hardware Advantages: SEGA Genesis games are more complex than Atari games due to superior hardware, including over 500 times more RAM, a wider range of control inputs, and improved graphics.
  • Generalization Potential: Genesis games often feature levels that share physics and object appearances but differ in layout and items, making them ideal for testing transfer learning.

Improvements over Previous Implementations

Gym Retro improves upon the Retro Learning Environment (RLE) and OpenAI's earlier "Universe" project:

  • Flexibility: Unlike RLE, which required C++ code for environment definitions, Gym Retro uses JSON files, simplifying the integration of new games.
  • Reliability: Gym Retro addresses the failures of the Universe project, which suffered from asynchronous environments, real-time constraints, and unreliable screen-based game state detection.

Supporting Resources for Researchers

To facilitate participation and development, OpenAI has provided several tools:

  • Retro-baselines: A GitHub repository showing how to run various RL algorithms on contest tasks.
  • Sonic Recordings: A dataset of human recordings beating the Sonic levels. These allow agents to start from random points in a level to explore areas they might not otherwise reach or to enable training via learning from demonstrations.
  • Technical Report: Detailed benchmark data and results are available in the accompanying "Gotta Learn Fast" paper.

Sources