Better exploration with parameter noise

Overview

Adding adaptive noise to the parameters of reinforcement learning algorithms frequently boosts performance, is simple to implement, and very rarely decreases performance, making it worth trying on any problem.

How Parameter Noise Works

Parameter noise injects randomness directly into the parameters of the agent’s neural network policy, altering the types of decisions it makes while keeping them fully dependent on current observations, unlike traditional action space noise which changes action likelihoods unpredictably. This approach sits between evolution strategies (which manipulate policy parameters without influencing actions during rollouts) and deep RL methods such as TRPO, DQN, and DDPG (which add noise to the action space but leave parameters untouched).

Empirical Results

After learning for 20 episodes on the HalfCheetah Gym environment, a policy trained with parameter noise achieves a score of around 3,000, whereas a policy trained with traditional action noise only reaches about 1,500. Parameter noise helps agents explore their environments more effectively, leading to higher scores and more elegant behaviors because exploration becomes consistent across timesteps.

Addressing Challenges

Three problems arose during research: different network layers have varying sensitivities to perturbations, the sensitivity of policy weights changes over time, and picking the right noise scale is difficult. Layer normalization was used to equalize layer sensitivities, ensuring perturbed layers stay within a similar distribution. An adaptive scheme adjusts perturbation size by measuring its effect on action space and comparing it to a target, moving the noise‑scale problem into the more interpretable action space.

Baselines and Benchmarks

Baseline code incorporating parameter noise has been released for DQN, Double DQN, Dueling DQN, Dueling Double DQN, and DDPG. Benchmarks compare DDQN with and without parameter noise on a subset of Atari games and three variants of DDPG on a range of Mujoco continuous‑control tasks.

Development Insights

Early experiments applying noise to the Q function of DQN sometimes caused the agent to repeat the same action; adding a separate policy head (as in DDPG) mitigated this. Later experiments showed that the separate head was unnecessary after improvements in noise rescaling, yielding a simpler, less costly implementation with comparable performance. This highlights how RL algorithms can fail silently, prompting engineers to build workarounds for undetected bugs.

Authors

Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Pieter Abbeel, Marcin Andrychowicz, Richard Chen, Xi Chen, Tasmin Asfour