OpenAI Retro Contest Results

TL;DR

OpenAI has completed the first run of the Retro Contest, a competition focused on developing reinforcement learning (RL) algorithms that can generalize from previous experience. The results demonstrate that the Sonic benchmark is an effective tool for testing general machine learning approaches, as top performance was achieved through the tuning and extension of established algorithms rather than competition-specific hacks.

Performance Benchmarks and Results

The top-performing agents in the contest achieved a maximum score of 4,692, which remains significantly below the theoretical maximum of 10,000. This gap indicates that there is substantial room for improvement in the ability of AI agents to generalize and learn quickly in new environments.

Top Scoring Teams

Rank	Team	Score
#1	Dharmaraja	4,692
#2	mistake	4,446
#3	aborg	4,430
#4	whatever	4,274
#5	Students of Plato	4,269
Baseline	Joint PPO	4,070
Baseline	Joint Rainbow	3,843
Baseline	Rainbow	3,498

Technical Approaches of Winning Solutions

The winning solutions relied on general machine learning techniques, specifically the refinement of Proximal Policy Optimization (PPO) and Rainbow DQN.

Dharmaraja (1st Place)

Team Dharmaraja utilized a variant of joint PPO with several key modifications:

Visual Input: Switched from grayscale to RGB images.
Action Space: Augmented the action space to include more common button combinations.
Reward Function: Implemented an augmented reward function that rewarded the agent for visiting new states, determined by a perceptual hash of the screen.

mistake (2nd Place)

Team mistake based their solution on the Rainbow baseline, training from scratch using Rainbow DQN. Their performance gains came from:

Hyper-parameter Tuning: Optimizing the value of $n$ for n-step Q learning.
Architecture: Adding an extra CNN layer to the model, which improved performance despite slowing down training.
Update Interval: Lowering the DQN target update interval.

aborg (3rd Place)

Team aborg used a variant of joint PPO, focusing on transfer learning and fast adaptation:

Pre-training: Utilized additional training levels from Game Boy Advance and Master System Sonic games.
Architecture: Modified the network architecture.
Hyper-parameter Optimization: Specifically tuned the learning rate to stabilize the first 150K timesteps of fine-tuning.

Evaluation Methodology

To prevent overfitting to the leaderboard, OpenAI employed a two-stage evaluation process:

Leaderboard Phase: Contestants received feedback via scores and videos based on a test set of five low-quality levels created with a level editor.
Final Evaluation: The top 10 entrants were tested against 11 custom Sonic levels designed by skilled level designers. Each agent was evaluated three times per level using different random seeds to reduce noise.

Key Lessons and Implications

OpenAI observed that the most successful approaches were not radically different from the internal benchmarks established by OpenAI prior to the contest. This highlighted two primary findings:

The Importance of Hyper-parameters: The top submissions showed that carefully tuned versions of existing baseline algorithms (like Rainbow DQN) can significantly outperform default settings.
Validation of the Sonic Benchmark: Because winning solutions were general ML approaches rather than specific hacks, OpenAI concluded that the Sonic benchmark is a robust problem for the community to solve.
Transfer Learning: The use of fine-tuning from pre-trained networks was a a successful strategy for several top teams.