OpenAI Procgen Benchmark Release

OpenAI has released the Procgen Benchmark, a suite of 16 procedurally-generated environments designed to measure reinforcement learning (RL) agents' ability to generalize skills to unseen levels. This benchmark addresses the common issue of overfitting in classic RL benchmarks, where agents may memorize specific trajectories rather than learning robust, generalizable skills.

Overview of the Procgen Benchmark

The Procgen Benchmark consists of 16 unique environments that allow for the generation of distinct training and test sets. This structure is an ideal for evaluating generalization, as it ensures that agents cannot simply memorize the environment. The benchmark is also designed to evaluate sample efficiency, as the diverse challenges within each environment require agents to learn robust policies to succeed.

Design Principles

All Procgen environments are built according to four key design principles:

• High Diversity: The environment generation logic is given maximal freedom to create a variety of levels, presenting agents with meaningful generalization challenges.
• Fast Evaluation: Environments are optimized to perform thousands of steps per second on a single CPU core. Baseline agents are calibrated to make significant progress after 200M timesteps.
• Tunable Difficulty: Two settings—easy and hard—are available. The hard setting is the standard for reporting results, while the easy setting requires approximately one-eighth of the resources to train.
• Visual Recognition and Motor Control: The environments mimic the style of Atari and Gym Retro games, focusing on the identification of key assets in the observation space and executing appropriate motor responses.

Evaluating Generalization and Overfitting

OpenAI conducted a study using Proximal Policy Optimization (PPO) on training sets ranging from 100 to 100,000 levels. The findings revealed several key insights into RL generalization:

• Overfitting to Small Sets: Agents strongly overfit to small training sets in almost all environments. In some cases, agents require as many as 10,000 levels to close the generalization gap.
• Implicit Curriculum: A trend emerged where training performance improved as training sets grew larger. This suggests that a larger, more diverse set of levels provides an implicit curriculum that helps the agent learn to generalize even within the training set itself.
• The Illusion of Progress: In an ablation study using deterministic levels, agents appeared to be making progress during training, but failed completely on test levels. This highlights the critical importance of using diverse environment distributions to avoid the illusion of progress caused by memorization.

Technical Implementation

For researchers and developers, the benchmark is available via pip install procgen. The OpenAI RL team uses this as their standard research platform. OpenAI found that all Procgen environments require training on 500–1000 different levels before agents can begin to generalize to new levels.