Stochastic Neural Networks for Hierarchical Reinforcement Learning
OpenAI has introduced a general framework for hierarchical reinforcement learning (HRL) that utilizes Stochastic Neural Networks to pre-train a diverse set of skills, which are then leveraged to accelerate learning in downstream tasks. This approach specifically addresses the challenges of sparse rewards and long-horizon tasks, where traditional deep reinforcement learning often struggles to explore effectively.
Pre-training Skills via Stochastic Neural Networks
The core of the framework is a pre-training phase where the agent learns useful skills in a pre-training environment. To efficiently learn a wide span of interpretable skills, the researchers employ Stochastic Neural Networks combined with an information-theoretic regularizer.
This pre-training process is guided by a single proxy reward, which requires minimal domain knowledge about the specific downstream tasks the agent will eventually face. By using an information-theoretic regularizer, the system ensures that the skills learned are distinct and cover a broad range of behaviors.
Hierarchical Learning Architecture
The framework operates on a hierarchical structure to improve exploration and sample efficiency:
- Skill Acquisition: The agent first learns a library of skills using the stochastic neural network architecture during the pre-training phase.
- High-Level Policy Training: Once the skills are acquired, a high-level policy is trained on top of these pre-existing skills. Instead of learning primitive actions, the high-level policy selects from the learned skills to achieve the goal.
This hierarchy allows the agent to tackle sparse rewards in downstream tasks more effectively because the high-level policy can explore the environment using complex, meaningful behaviors (skills) rather than random primitive actions.
Performance and Implications
Experiments demonstrate that the combination of Stochastic Neural Networks and information-theoretic regularization is effective for learning interpretable skills in a sample-efficient manner. The researchers found that this approach significantly boosts learning performance uniformly across a wide range of downstream tasks.
By combining the strengths of intrinsic motivation and hierarchical methods, this framework provides a scalable way to learn complex behaviors without requiring extensive manual engineering of reward functions for every new task.