OpenAI Evolved Policy Gradients (EPG)

OpenAI has introduced Evolved Policy Gradients (EPG), a metalearning method that evolves loss functions rather than policies to enable reinforcement learning (RL) agents to learn new tasks more efficiently. This approach allows agents to avoid starting from a "blank slate" by leveraging evolved loss functions that encode a sense of progress across similar tasks.

Evolving Loss Functions for Metalearning

EPG focuses on evolving the loss function itself to improve learning efficiency and generalization. This differs from other metalearning approaches that focus on learning recurrent policies or policy initializations. OpenAI notes that recurrent policies often overfit the specific task and policy initializations have limited expressivity regarding exploration.

Loss functions are hypothesized to be the objects most likely to generalize across substantially different tasks. This is based on the same logic as hand-engineered loss functions, such as those used in Proximal Policy Optimization (PPO), which are generically applicable across diverse problems like Atari games and robotic control.

Generalization Performance and the "Ant" Experiment

To test the generalization capabilities of EPG, OpenAI conducted an experiment using agents ("ants") tasked with walking to randomly located targets on the right half of an arena.

Key results from the experiment include:

• Training: The EPG loss was evolved to be effective for targets on the right half of the arena.
• Testing: After freezing the loss, the agents were given targets on the left half of the arena—a task outside the training distribution.
• Outcome: The agents successfully learned to walk to the left, demonstrating generalization to tasks outside the training distribution.

In comparison, the RL2 algorithm, which learns a policy that adapts to novel tasks, failed in this scenario. While RL2 succeeded on the right-half targets, it qualitatively failed when presented with left-half targets, continuing to walk to the right. This suggests that RL2 overfit to the specific training tasks.

Current Limitations and the Trade-off Between Performance and Generality

While EPG shows promise in generalization within a task family, it currently faces a trade-off between performance and the generality of the loss function.

• Task Family Limitation: EPG can currently only train a loss function effective for one small family of tasks (e.g., walking left and right).
• Comparison to Standard RL: Standard RL loss functions are highly general; the same loss can be used for vastly different skills, such as playing Space Invaders. EPG loss functions, however, are not yet general enough to be cross-applicable to wildly different tasks.

OpenAI concludes that EPG gains performance within specific task families by sacrificing the broad generality found in standard RL methods.