OpenAI Learning a Hierarchy Research

OpenAI has introduced Meta-Learning Shared Hierarchies (MLSH), a hierarchical reinforcement learning algorithm designed to solve tasks requiring thousands of timesteps by automatically discovering high-level actions. This approach allows agents to master new, complex navigation tasks quickly by sequencing learned components rather than relying on brute-force search over low-level actions.

The Challenge of Long-Horizon Tasks

Traditional reinforcement learning methods often rely on brute-force search over low-level actions, which becomes highly inefficient when a task requires a large number of timesteps. For example, a solution that requires 2,000 low-level actions is computationally expensive to solve through simple trial and error.

MLSH addresses this by representing complicated behaviors as short sequences of high-level actions. By reducing a 2,000-step low-level sequence into a 10-step high-level sequence, the agent can search for a solution much more efficiently.

Meta-Learning Shared Hierarchies (MLSH) Architecture

MLSH implements a hierarchical policy consisting of a master policy and a set of sub-policies. The master policy selects a high-level action every $N$ timesteps (where $N$ may be 200), and the chosen sub-policy is then executed for the duration of those $N$ timesteps.

Key technical characteristics of MLSH include:

• Automatic Discovery: Unlike previous hierarchical policies that were hand-engineered, MLSH discovers its hierarchical structure automatically through interaction with the environment.
• Meta-Learning Perspective: The algorithm defines a "good" hierarchy as one that enables an agent to reach high rewards quickly when training on previously unseen tasks.
• Shared Sub-policies: The agent is trained on a distribution of tasks. While the master policy is learned anew for each sampled task, the sub-policies are shared across tasks, allowing the agent to refine a set of general-purpose high-level actions.

Experimental Results in AntMaze

To evaluate the algorithm, OpenAI researchers used the AntMaze environment, featuring a MuJoCo Ant robot navigating through a distribution of nine different mazes.

Through interaction with the environment, MLSH successfully discovered a diverse set of sub-policies—specifically, walking and crawling in different directions—which could be sequenced together to solve the maze tasks. The researchers noted that this set of sub-policies enabled the agent to master tasks larger than those it was originally trained on.

Availability

OpenAI has released the code for training MLSH agents and the MuJoCo environments used for the evaluation of these algorithms on GitHub.