OpenAI Random Network Distillation (RND) for Reinforcement Learning

OpenAI has developed Random Network Distillation (RND), a prediction-based method to encourage reinforcement learning (RL) agents to explore environments through curiosity. This approach allows agents to visit unfamiliar states by rewarding them based on the difficulty of predicting the output of a fixed random neural network. For the first time, this method has exceeded average human performance on the Atari game Montezuma's Revenge, achieving state-of-the-art performance and occasionally solving the first level by finding all 24 rooms without relying on human demonstrations or access to the underlying emulator state.

Random Network Distillation (RND) Technical Overview

Random Network Distillation is designed to incentivize the exploration of unfamiliar states. The system uses a fixed, randomly initialized neural network as a target. A second network, the predictor, attempts to predict the output of this target network given the next state.

Because the predictor network is trained on the states the agent visits, its prediction error is lower in frequently visited states and higher in novel states. This prediction error serves as an intrinsic reward, driving the agent toward unexplored areas of the environment.

Solving the Noisy-TV Problem

Previous curiosity-driven methods based on next-state prediction were susceptible to the "noisy-TV problem," where agents become trapped by stochastic elements of the environment (such as a TV playing random static) because they are perpetually unpredictable and thus yield high intrinsic rewards.

RND eliminates this problem by focusing on predicting the output of a deterministic synthetic problem (the fixed random network) rather than the next state of the environment itself. This ensures that prediction errors are caused by novelty rather than stochasticity, making RND immune to the noisy-TV problem.

Performance in Montezuma's Revenge

Montezuma's Revenge is widely considered a benchmark for exploration in RL because simple exploration strategies typically fail to gather rewards. In the original DQN paper, it was the only game where the agent achieved 0% of the average human score (4.7K).

Using RND, OpenAI achieved the following results:

  • Large-scale experiment: With 1024 rollout workers, the agent achieved a mean return of 10K over 9 runs, with a best mean return of 14.5K. Most runs discovered between 20 and 22 rooms.
  • Long-running experiment: One run out of 10 achieved a best return of 17.5K, which corresponds to passing the first level and finding all 24 rooms.

Curiosity-Driven Learning Across Environments

OpenAI conducted a large-scale study across 50+ different environments without any environment-specific (extrinsic) rewards. The agents demonstrated varying levels of competence:

  • Super Mario Bros: The agent discovered 11 levels, found secret rooms, and defeated bosses, as the intrinsic reward aligned well with the game's objective of advancing.

  • Bowling: The agent outperformed agents trained on clipped extrinsic rewards, likely because it was attracted to the unpredictable flashing of the scoreboard after strikes.

  • Pong: The agent learned to keep the ball in play to prolong rallies, though it prioritized prolonging the game over winning.

  • Breakout: The agent experienced spikes of intrinsic reward when encountering new brick configurations or passing a level for the first time.

Implementation and Training Stability

OpenAI notes that implementation details are critical for the success of RND. Stability was achieved by avoiding feature saturation and bringing intrinsic rewards into a predictable range. The team found that significant performance improvements occurred each time a bug was fixed, emphasizing that simpler algorithms are often preferable for easier debugging and stability.

To integrate the exploration bonus with extrinsic rewards, OpenAI used a variant of Proximal Policy Optimization (PPO) with two separate value heads for the two reward streams. This allows for different discount rates and the combination of episodic and non-episodic returns.

Sources