Proximal Policy Optimization (PPO) Release Notes – OpenAI Baselines

TL;DR

OpenAI announced the release of Proximal Policy Optimization (PPO), a new class of reinforcement learning algorithms that performs comparably or better than state-of-the-art approaches while being much simpler to implement and tune, and has become the default RL algorithm at OpenAI.

Why PPO Matters

Policy gradient methods are sensitive to stepsize choices and often require millions or billions of timesteps to learn simple tasks. Existing constrained update methods such as TRPO and ACER address these issues but introduce complexity: ACER needs off‑policy corrections and a replay buffer, and TRPO is not easily compatible with shared‑parameter architectures. PPO offers a balance of ease of implementation, sample efficiency, and tuning simplicity.

Core Idea of PPO

PPO seeks to compute a policy update that improves the objective while keeping the deviation from the previous policy small. The algorithm uses a clipped surrogate objective that enables trust‑region‑style updates compatible with stochastic gradient descent.

Clipped Objective Formula

The objective used in PPO is:

$$L^{C L I P} \left(\theta \right) = \left(\hat{E}\right){t} \left[ \min \left(\ r{t} \left(\theta \right) \ \hat{A}{t},\ \text{clip}\left(r{t} \left(\theta \right), 1-\epsilon, 1+\epsilon\right) \hat{A}_{t} \right) \right]$$

where:

• (\theta) are the policy parameters
• (\left(\hat{E}\right)_{t}) is the empirical expectation over timesteps
• (r_{t}) is the ratio of the probability under the new and old policies
• (\left(\hat{A}\right)_{t}) is the estimated advantage at time (t)
• (\epsilon) is a hyperparameter, typically 0.1 or 0.2

This formulation implements a trust region update without a KL penalty, simplifying the algorithm while maintaining performance.

Empirical Performance

In tests, the clipped PPO objective displayed the best performance on continuous control tasks and almost matched ACER’s performance on Atari benchmarks, despite being far simpler to implement.

Baselines Release

The OpenAI Baselines repository now includes scalable, parallel implementations of PPO and TRPO that use MPI for data passing, both written for Python 3 and TensorFlow. Pre‑trained policies for the Roboschool agent zoo are also provided.

PPO2 and ACER Additions

An update introduced a GPU‑enabled implementation called PPO2, which runs approximately three times faster than the original PPO baseline on Atari. Additionally, an implementation of Actor‑Critic with Experience Replay (ACER) was released; ACER uses a replay buffer to perform multiple gradient updates per experience sample and incorporates a Q‑function approximator trained with the Retrace algorithm.

Controllable Robots Demonstration

Interactive agents trained with PPO were built in Roboschool. Using a keyboard, users can set new target positions for a robot; although the input sequences differ from those seen during training, the policies generalize to the new commands.

Call for Contributors

OpenAI is seeking contributors to help build and optimize the reinforcement learning codebase. Interested individuals are invited to apply via the provided link and mention that they read the baselines PPO post in their application.