OpenAI Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines

OpenAI has introduced a method for variance reduction in policy gradient methods using action-dependent factorized baselines. This approach reduces the high variance of gradient estimates, which is a critical bottleneck in deep reinforcement learning, particularly for problems with long horizons or high-dimensional action spaces.

Action-Dependent Baselines for Variance Reduction

Policy gradient methods often suffer from high variance in their gradient estimates, which can lead to unstable learning and slow convergence. To address this issue, OpenAI researchers have derived a bias-free action-dependent baseline that leverages the structural form of the stochastic policy itself.

Unlike traditional state-dependent baselines, this method does not make additional assumptions about the Markov Decision Process (MDP) and remains bias-free. The researchers demonstrate that action-dependent baselines can outperform optimal state-dependent baselines, which are suboptimal in high-dimensional control problems.

Performance in High-Dimensional Control

The proposed action-dependent baseline method is computationally efficient and scales effectively to high-dimensional control problems. The researchers validated this efficiency through a synthetic 2000-dimensional target matching task, where the algorithm successfully scaled to such high dimensions.

Experimental results across standard reinforcement learning benchmarks, high-dimensional hand manipulation tasks, and synthetic tasks indicate that the agent learns faster when using action-dependent baselines.

Extensions to Partially Observed and Multi-Agent Tasks

The general principle of incorporating additional information into baselines for improved variance reduction can be extended beyond standard MDPs. The researchers show that this approach can be applied to partially observed tasks and multi-agent reinforcement learning environments, providing a further increase in stability and stability in complex scenarios.

Summary of Technical Contributions

  • Bias-Free Variance Reduction: A new baseline derivation that exploits the policy's structural form to reduce gradient variance without introducing bias.
  • Scalability: Demonstrated effectiveness in high-dimensional action spaces (up to 2000 dimensions) in synthetic tasks.
  • Learning Speed: Faster convergence on standard RL benchmarks and complex hand manipulation tasks.
  • Generalizability: Applicability to extension to partially observed and multi-agent reinforcement learning.

Sources