Quantifying Generalization in Reinforcement Learning

OpenAI has released CoinRun, a training environment designed to provide a precise metric for an agent's ability to generalize experience to novel situations. This environment addresses a longstanding challenge in reinforcement learning (RL), where agents frequently overfit to the specifics of their training environments rather than learning generalizable skills.

The Generalization Challenge in RL

Deep reinforcement learning algorithms often struggle to transfer experience to new environments. While agents can solve complex tasks, they frequently overfit by latching onto specific environmental details. This issue is part of a broader pattern where RL agents are often benchmarked on the same environments they were trained on, which the researchers compare to testing on a training set in supervised learning.

Previous attempts to address this using the Sonic benchmark, procedurally generated gridworld mazes, and the General Video Game AI framework showed similar patterns of overfitting. For example, agents trained on the Sonic benchmark performed well on training levels but poorly on test levels without fine-tuning.

The CoinRun Environment

CoinRun is a procedurally generated platformer environment designed to be more tractable than complex games like Sonic the Hedgehog while still providing a significant generalization challenge.

Rules and Objectives

  • Goal: The agent must collect a single coin located at the end of the level.
  • Obstacles: The level contains both stationary and non-stationary obstacles; colliding with any obstacle results in immediate death.
  • Reward Structure: A fixed positive constant reward is granted only upon collecting the coin.
  • Termination: An episode ends when the agent dies, collects the coin, or reaches 1,000 time steps.

Evaluating Generalization Performance

OpenAI evaluated nine agents using Proximal Policy Optimization (PPO) for 256 million timesteps. Eight agents were trained on fixed sets of levels ranging from 100 to 16,000, while one agent was trained on an unrestricted set of levels where it never encountered the same level twice.

Key Findings on Overfitting

  • Data Volume: Substantial overfitting occurs when there are fewer than 4,000 training levels. Overfitting persists even with 16,000 training levels.
  • Optimal Performance: Agents trained with the unrestricted set of levels performed the best, as they had access to the most data (approximately 2 million distinct levels).
  • Architecture Impact: Agents using the IMPALA-CNN architecture generalized significantly better than those using the Nature-CNN baseline across all training set sizes.

Techniques for Improving Generalization

Using a fixed training set of 500 levels, OpenAI investigated several regularization techniques to reduce the generalization gap:

  • Environmental Stochasticity: This technique improved generalization to a greater extent than any other method tested.
  • Data Augmentation and Batch Normalization: Both significantly improved generalization performance.
  • L2 Regularization and Dropout: Both reduced the generalization gap, with L2 regularization having a more significant impact.

Extensions: CoinRun-Platforms and RandomMazes

To further investigate overfitting, OpenAI developed two additional environments:

  • CoinRun-Platforms: A variant where the agent must collect multiple coins scattered across platforms in a larger, fixed-size level, requiring more active exploration and memory.
  • RandomMazes: A simple maze navigation environment.

In both cases, agents strongly overfit. In RandomMazes, a significant generalization gap remained even when the agents were trained on 20,000 levels, necessitating the use of an IMPALA-CNN architecture followed by an LSTM for memory.

Future Research Directions

OpenAI suggests three primary areas for future research to improve RL generalization:

  1. Investigating the relationship between environment complexity and the number of levels required for effective generalization.
  2. Determining if different recurrent architectures are better suited for generalization.
  3. Exploring the most effective ways to combine different regularization methods.

Sources