Advantage Actor Critic (A2C) Explained

TL;DR

Advantage Actor Critic (A2C) is a hybrid reinforcement learning architecture that combines policy-based (Actor) and value-based (Critic) methods to reduce the high variance associated with pure policy gradient methods like Reinforce. This combination stabilizes training and improves sample efficiency by using a value function to provide immediate feedback on actions rather than relying on full-episode Monte Carlo returns.

The Variance Problem in Reinforce

Pure policy-based methods, such as Reinforce, suffer from significant variance in policy gradient estimation because they rely on Monte Carlo sampling. In Reinforce, the agent collects an entire trajectory and uses the discounted return to increase or decrease the probability of every action taken in that trajectory.

While this approach is unbiased because it uses true returns, it is highly unstable. Stochasticity in the environment and the policy means that the same starting state can lead to vastly different returns across episodes. To mitigate this variance, practitioners must use a large number of trajectories, which significantly reduces sample efficiency and slows down training.

The Actor-Critic Architecture

Actor-Critic methods resolve the variance problem by combining two distinct function approximations:

  • The Actor: A policy function $\pi_{\theta}(s, a)$ that controls how the agent behaves.
  • The Critic: A value function $(\hat{q})_{w}(s, a)$ that measures the quality of the action taken.

The Actor-Critic Training Process

During training, the Actor and Critic optimize together through a continuous feedback loop:

  1. Action Selection: The Actor receives the current state $S_t$ and outputs an action $A_t$.
  2. Value Estimation: The Critic receives $S_t$ and $A_t$ and computes the Q-value (the value of taking that action in that state).
  3. Environment Interaction: The action $A_t$ is performed, resulting in a new state $S_{t+1}$ and a reward $R_{t+1}$.
  4. Policy Update: The Actor updates its policy parameters based on the Q-value provided by the Critic.
  5. Value Update: The Critic updates its own value parameters to improve its future feedback.

Advantage Actor Critic (A2C)

Advantage Actor Critic (A2C) further stabilizes learning by replacing the standard action-value function with the Advantage function.

Instead of measuring the absolute value of an action, the Advantage function calculates how much better a specific action is compared to the average value of that state. It subtracts the mean value of the state from the state-action pair:

$$\text{Advantage} = Q(s, a) - V(s)$$

Implementation via TD Error

Implementing a true advantage function typically requires two separate value functions ($Q(s, a)$ and $V(s)$). To simplify this, A2C uses the Temporal Difference (TD) error as a reliable estimator of the advantage function.

If the calculated advantage $A(s, a)$ is positive, the gradient is pushed in that direction to increase the action's probability. If $A(s, a)$ is negative—meaning the action performed worse than the average for that state—the gradient is pushed in the opposite direction.

Practical Application in Robotics

Hugging Face demonstrates the application of A2C using the Stable-Baselines3 library within PyBullet robotics simulations. This allows for the training of agents in complex continuous control environments, such as teaching a bipedal walker or a spider robot to walk.

Sources