OpenAI Baselines: ACKTR & A2C

OpenAI has released two new implementations for its Baselines framework: ACKTR (Actor Critic using Kronecker-factored Trust Region) and A2C (Advantage Actor Critic). These releases provide researchers with tools that improve sample efficiency in reinforcement learning and offer a more computationally efficient, synchronous alternative to the asynchronous A3C algorithm.

ACKTR: High Sample Efficiency and Scalability

ACKTR is designed to be more sample-efficient than first-order methods like A2C and other trust region methods like TRPO. It achieves this by taking steps in the natural gradient direction rather than the standard gradient direction.

Technical Foundation

ACKTR combines three core techniques to optimize policy improvement:

  • Actor-Critic Methods: Combining policy-based and value-based learning.
  • Trust Region Optimization: Ensuring consistent improvement by limiting the KL divergence to prevent the new policy from behaving radically differently than the old one, which avoids performance collapse.
  • Distributed Kronecker Factorization: Used to improve both scalability and sample efficiency.

Sample vs. Computational Complexity

ACKTR optimizes the trade-off between the number of interaction timesteps (sample complexity) and the number of numerical operations (computational complexity):

  • Sample Complexity: ACKTR has better sample complexity than first-order methods because the natural gradient identifies the direction in parameter space that achieves the largest instantaneous improvement in the objective per unit of change in the output distribution.
  • Computational Complexity: The KFAC update used by ACKTR is only 10–25% more expensive per update step than a standard gradient update. This makes it significantly more efficient than TRPO, which requires expensive conjugate-gradient computations.

Performance and Scaling

ACKTR performance scales well with batch size. Because it approximates the local curvature of the parameter space using information from each batch, it is particularly effective for large-scale distributed training where large batch sizes are common.

A2C: A Synchronous Alternative to A3C

A2C is a synchronous, deterministic variant of the Asynchronous Advantage Actor Critic (A3C) algorithm. While A3C was influential, OpenAI found that the asynchrony in A3C did not provide a performance benefit.

A2C vs. A3C

  • Mechanism: Unlike A3C, which updates asynchronously, A2C waits for each actor to finish its segment of experience before performing an update, averaging over all actors.

  • Performance: OpenAI's synchronous A2C implementation performs better than their asynchronous implementations. They found no evidence that the noise introduced by asynchrony provides any regularization or exploration benefit.

  • Efficiency: A2C is more cost-effective on single-GPU machines because it can more effectively utilize GPUs with large batch sizes. It is also faster than CPU-only A3C implementations when using larger policies.

Benchmarks and Applications

OpenAI released benchmarks evaluating ACKTR and A2C against PPO and ACER on a range of tasks. ACKTR was tested on 49 Atari games, with hyperparameters tuned solely on the game Breakout.

ACKTR has been applied to learn control policies for:

  • Simulated robots using pixels as input and continuous action spaces.
  • Atari agents using pixels as input and discrete action spaces.

Sources