OpenAI Variational Option Discovery Algorithms

OpenAI has introduced new methods for option discovery based on variational inference, specifically proposing the Variational Autoencoding Learning of Options by Reinforcement (VALOR) algorithm and a curriculum learning strategy to improve the discovery of diverse behavioral modes.

Variational Autoencoding Learning of Options by Reinforcement (VALOR)

VALOR is a reinforcement learning method derived from a tight connection between variational option discovery and variational autoencoders (VAEs). In this framework, the agent's policy acts as an encoder that maps contexts from a noise distribution into specific trajectories. Conversely, the decoder is designed to recover those original contexts from the complete trajectories generated by the agent.

Curriculum Learning for Option Stability

To address training instability in variational option discovery, OpenAI proposes a curriculum learning approach. The number of contexts the agent is exposed to increases only when the agent's performance on the current set of contexts is sufficiently strong, as measured by the decoder.

This curriculum-based approach provides two primary benefits:

• Training Stabilization: It stabilizes the training process for VALOR and other existing variational option discovery methods.
• Increased Behavioral Diversity: It allows a single agent to learn significantly more modes of behavior than is possible when using a fixed context distribution.

Scope of Research

Beyond the primary algorithmic contributions, the research also investigates the fundamental limitations of the general variational option discovery approach and examines how learned options can be applied to downstream tasks.