OpenAI Better Exploration with Parameter Noise

OpenAI has demonstrated that adding adaptive noise to the parameters of reinforcement learning (RL) algorithms frequently improves performance and accelerates the learning process. This technique provides a more consistent form of exploration than traditional action-space noise, making it a versatile addition to various RL problems.

Parameter Noise vs. Action Space Noise

Parameter noise injects randomness directly into the parameters of the agent's neural network policy, ensuring that the agent's decisions remain fully dependent on its current sensory input. This differs from traditional RL exploration in the following ways:

  • Action Space Noise: Traditional RL changes the likelihood of actions from one moment to the next, leading to unpredictable exploration that is not correlated with the agent's specific parameters.
  • Parameter Space Noise: By altering the parameters themselves, the agent's exploration becomes consistent across different timesteps, leading to more elegant behaviors and higher scores.

This approach serves as a middle ground between evolution strategies, which manipulate policy parameters without influencing actions during a rollout, and deep RL approaches like TRPO, DQN, and DDPG, which typically add noise to the action space.

Performance Gains in Continuous Control

Parameter noise allows agents to master tasks more rapidly than traditional methods. In the HalfCheetah Gym environment, a policy trained with parameter noise achieved a score of approximately 3,000 after 20 episodes, whereas a policy using traditional action noise only reached a score of around 1,500.

Technical Implementation and Challenges

OpenAI identified three primary challenges when applying parameter noise to deep neural networks and implemented specific technical solutions for each:

1. Layer Sensitivity

Different network layers exhibit different sensitivities to perturbations. OpenAI utilized layer normalization to ensure that the output of a perturbed layer remains within a similar distribution before becoming the input for the next layer.

2. Weight Sensitivity and Predictability

The sensitivity of policy weights can shift during training, making action prediction difficult. OpenAI introduced an adaptive scheme to adjust the size of parameter space perturbations by measuring their effect on the action space.

3. Noise Scale Selection

Choosing the correct noise scale in parameter space is unintuitive. The adaptive scheme solves this by pushing the problem of choosing the noise scale into the action space, which is more interpretable for researchers.

Algorithm Compatibility and Baselines

OpenAI has extended the use of parameter noise to both on-policy and off-policy algorithms. Baseline code incorporating this technique has been released for the following algorithms:

  • DQN
  • Double DQN
  • Dueling DQN
  • Dueling Double DQN
  • DDPG

Benchmarks for these implementations include DDQN performance on a subset of Atari games and three DDPG variants across various continuous control tasks in the Mujoco simulator.

Development Insights on DQN Implementation

During early development, OpenAI initially added a separate policy head to DQN to prevent extreme perturbations from causing the algorithm to repeat the same action. However, subsequent experiments revealed that this separate head was unnecessary. By refining how noise was re-scaled, the team achieved similar results with a simpler, less costly implementation. OpenAI notes that this underscores a common issue in reinforcement learning where algorithms can "fail silently and subtly," potentially leading developers to engineer complex solutions for bugs that could be solved more simply.

Sources