Adversarial Attacks on Neural Network Policies
OpenAI researchers have demonstrated that neural network policies used in reinforcement learning (RL) are susceptible to adversarial attacks, similar to vulnerabilities previously identified in computer vision classifiers. By introducing small, carefully crafted perturbations to raw input data, adversaries can significantly degrade the test-time performance of trained policies without interfering with human perception.
Vulnerability of Reinforcement Learning Policies
Neural network policies in reinforcement learning are vulnerable to adversarial examples—inputs specifically constructed to force a model into making incorrect decisions. While this phenomenon was well-documented in machine learning classifiers for image recognition, this research confirms that the same vulnerabilities extend to the decision-making policies of RL agents.
Regardless of the learned task or the specific training algorithm used, the researchers observed a significant drop in agent performance when subjected to adversarial perturbations. This indicates a systemic vulnerability in how neural network policies process input data to determine actions.
Threat Model and Attack Methodology
The research utilizes a threat model where the adversary is capable of introducing small perturbations to the raw input of the policy. These perturbations are designed to be minimal enough that they do not alter the input in a way that is perceptible to humans, yet they are sufficient to mislead the neural network.
Attack Settings
The study characterized the degree of vulnerability across different scenarios:
- White-box settings: The adversary has full knowledge of the policy's internal parameters and architecture.
- Black-box settings: The adversary has limited or no knowledge of the internal workings of the policy.
In both settings, existing adversarial example crafting techniques were effective in degrading the performance of the trained policies.
Implications for AI Safety
The finding that RL policies can be easily disrupted by imperceptible input changes suggests a critical security gap in the deployment of autonomous agents. Because these attacks work across various tasks and training algorithms, the vulnerability appears to be a fundamental characteristic of the neural network architectures currently used for reinforcement learning, rather than a flaw in a specific implementation.