OpenAI Large-scale Study of Curiosity-Driven Learning
OpenAI has released a study on curiosity-driven learning, demonstrating that agents can learn complex behaviors and solve environments without any external rewards. By using prediction error as an intrinsic reward signal, these agents exhibit a high degree of alignment between curiosity-driven objectives and traditional hand-designed extrinsic rewards.
Intrinsic Curiosity as a Reward Signal
Reinforcement learning (RL) typically relies on extrinsic rewards engineered by humans to guide an agent's behavior. However, creating dense, hand-designed rewards for every environment is not scalable. To address this problem, the research team developed a reward function intrinsic to the agent, where curiosity is defined as the prediction error of the agent's internal model of the environment.
The agent is rewarded when it encounters states or transitions that its internal model cannot accurately predict, encouraging exploration of novel or unfamiliar territory.
Large-Scale Evaluation Across 54 Environments
OpenAI conducted the first large-scale study of purely curiosity-driven learning across 54 standard benchmark environments, including the Atari game suite. The agents were trained without any extrinsic rewards, meaning they did not receive signals such as game scores or points.
The results showed surprisingly good performance, and the research found a high degree of alignment between the intrinsic curiosity objective and the extrinsic rewards typically used in these games.
Feature Space and Generalization
The research investigated how different feature spaces are used to compute prediction error. The study found that:
Random Features: These are sufficient for many popular RL game benchmarks.
Learned Features: These provide better generalization, specifically when the agent is tasked with navigating novel game levels in Super Mario Bros.
Limitations in Stochastic Environments
While curiosity-driven learning is successful in many benchmarks, the researchers identified limitations of prediction-based rewards in stochastic (random) setups. In environments where transitions are inherently unpredictable, the agent may become trapped by the unpredictable nature of the ruangan, rather than seeking meaningful exploration.
Conclusion
This research demonstrates that curiosity-driven learning can effectively replace extrinsic rewards in many cases, and the agent's ability to generalize to new levels is improved when using learned features. However, the challenges of stochasticity remain a primary limitation for prediction-based intrinsic rewards.