Equivalence between policy gradients and soft Q-learning

OpenAI researchers have established a precise mathematical equivalence between "soft" (entropy-regularized) Q-learning and policy gradient methods. This finding provides a theoretical explanation for why Q-learning methods often remain effective despite estimating inaccurate Q-values, suggesting that these methods may be implicitly implementing policy gradient updates.

Theoretical Equivalence in Entropy-Regularized RL

Soft Q-learning is exactly equivalent to a policy gradient method when applied within the framework of entropy-regularized reinforcement learning. This equivalence suggests that the two traditionally distinct approaches to model-free reinforcement learning—Q-learning (which focuses on value estimation) and policy gradients (which focus on direct policy optimization)—converge under specific regularization conditions.

Connection to Natural Policy Gradients

The research identifies a further connection between standard Q-learning methods and natural policy gradient methods. This link helps contextualize how value-based methods relate to more advanced policy optimization techniques that account for the geometry of the policy space.

Experimental Validation and Performance

OpenAI tested entropy-regularized versions of both Q-learning and policy gradients on the Atari benchmark, yielding the following results:

  • Performance: Entropy-regularized variants performed as well as, or slightly better than, the standard versions of these algorithms.
  • Practical Application: The researchers successfully constructed a Q-learning method that closely matched the learning dynamics of A3C (Asynchronous Advantage Actor-Critic). This specific implementation achieved these results without requiring a target network or an $\epsilon$-greedy exploration schedule.

Implications for Model-Free RL

This work addresses a long-standing gap in the understanding of Q-learning's effectiveness. Because Q-values are often empirically inaccurate, the theoretical bridge to policy gradients explains how Q-learning can still drive successful policy improvement. By proving that soft Q-learning is a form of policy gradient, the research simplifies the conceptual landscape of reinforcement learning algorithms.

Sources