OpenAI Plan Online Learn Offline (POLO) Framework
OpenAI has introduced the Plan Online Learn Offline (POLO) framework, a method for agents with internal models to continually act and learn within an environment. This framework is designed to increase learning efficiency and exploration stability in complex control tasks by creating a synergistic relationship between local trajectory optimization and global value function learning.
Synergistic Model-Based Control and Value Learning
The POLO framework operates on the principle that local model-based control and global value function learning can mutually reinforce each other to overcome their individual limitations.
Local Trajectory Optimization
Local trajectory optimization allows an agent to cope with approximation errors in the global value function. By planning locally, the agent can stabilize and accelerate the learning process of the value function.
Global Value Function Learning
Approximate value functions provide a global perspective that helps the agent reduce its planning horizon. This reduction in the horizon allows the agent to discover and implement policies that extend beyond simple local solutions.
Temporally Coordinated Exploration
Efficient learning in complex environments requires the agent to explore its state space effectively. POLO utilizes trajectory optimization to perform temporally coordinated exploration.
This exploration strategy is implemented by combining trajectory optimization with the estimation of uncertainty in the value function approximation. This coordinated approach is critical for the fast and stable learning of the value function.
Performance in Complex Control Tasks
The POLO framework enables the solution of complex simulated control tasks with high sample efficiency. Specifically, the framework was applied to:
- Humanoid Locomotion: Enabling a simulated humanoid to walk.
- Dexterous In-Hand Manipulation: Managing complex object manipulation within a simulated hand.
According to OpenAI, these tasks were solved in the equivalent of a few minutes of experience in the real world.