Evolution Strategies as a Scalable Alternative to Reinforcement Learning

OpenAI has discovered that Evolution Strategies (ES), a decades-old optimization technique, performs competitively with standard reinforcement learning (RL) on modern benchmarks such as Atari and MuJoCo. ES overcomes several traditional RL inconveniences by eliminating the need for backpropagation, simplifying distributed scaling, and improving robustness against sparse rewards.

Evolution Strategies vs. Reinforcement Learning

Evolution Strategies (ES) function as a black-box stochastic optimization technique. Unlike traditional RL, which optimizes a policy by injecting noise into the action space and using backpropagation to update parameters, ES injects noise directly into the parameter space.

The ES Algorithm

ES treats the policy network as a black box where a set of parameters (weights) goes in and a single total reward comes out. The optimization process follows a "guess and check" cycle:

  1. Perturbation: The algorithm takes a parameter vector and generates a population of slightly different versions by adding Gaussian noise.
  2. Evaluation: Each candidate is run independently in the environment to calculate the total reward.
  3. Update: The parameter vector is updated as a weighted sum of the candidates, where weights are proportional to the rewards achieved.

This process is mathematically equivalent to estimating the gradient of the expected reward in the parameter space using finite differences along random directions.

Technical Advantages of ES

ES provides several operational advantages over traditional RL algorithms:

  • Elimination of Backpropagation: Because ES only requires the forward pass of the policy, it does not require backpropagation or value function estimation. This results in code that is 2-3 times faster in practice and allows for the use of non-differentiable policies, such as binary networks or complex modules like pathfinding.
  • High Parallelizability: ES workers only need to communicate a few scalars (the rewards) rather than synchronizing entire parameter vectors. By controlling random seeds, workers can locally reconstruct perturbations. This enables linear speedups when scaling to thousands of CPU cores.
  • Increased Robustness: ES is less sensitive to hyperparameters that often cause RL to fail. For example, ES maintains consistent performance across different frame-skip settings in Atari games, whereas RL is not "scale-free."
  • Consistent Exploration: By using deterministic policies, ES avoids the "random jitter" often seen in policy gradient methods, allowing for more consistent exploration of the environment.
  • Long-term Credit Assignment: ES is particularly effective when episodes have many time steps, when actions have long-lasting effects, or when reliable value function estimates are unavailable.

Performance and Scalability Benchmarks

OpenAI compared ES against standard RL benchmarks, specifically MuJoCo control tasks and Atari games, focusing on data efficiency and wall-clock time.

MuJoCo Control Tasks

While ES is less data-efficient than TRPO (by a factor of approximately 10), it is significantly faster in terms of wall-clock time due to its scalability. Using 1,440 CPUs across 80 machines, ES trained a 3D MuJoCo humanoid walker in 10 minutes, compared to approximately 10 hours for A3C using 32 cores.

Atari Game Playing

Using 720 cores, ES achieved performance comparable to A3C on Atari games, reducing the training time from one day (for A3C on 32 cores) to just one hour.

Practical Limitations and Scope

ES is not a universal replacement for all machine learning techniques. OpenAI noted two primary limitations:

  1. Parameter Sensitivity: For ES to generate a gradient signal, adding noise to parameters must lead to different behavioral outcomes. OpenAI found that virtual batchnorm helps alleviate this, but further research into network parameterization is required.
  2. Sparse Reward Challenges: In environments like Montezuma’s Revenge, where specific sequences of actions are required to obtain a reward (e.g., finding a key), random actions in RL can occasionally succeed where random parameter noise in ES may not.

Note on Supervised Learning: ES is not intended for supervised learning tasks (e.g., image classification). In tests on the MNIST digit recognition task, ES was up to 1,000 times slower than backpropagation because supervised learning allows for the computation of exact gradients, making the sampling-based approach of ES inefficient.

Sources