Attacking Machine Learning with Adversarial Examples

TL;DR

Adversarial examples are intentionally designed inputs that cause machine learning models to make mistakes, acting as "optical illusions for machines." Their existence demonstrates that even simple modern algorithms can behave in ways designers do not intend, representing a concrete and urgent challenge for AI safety.

Understanding Adversarial Examples

Adversarial examples are inputs specifically crafted to trick a machine learning model into misclassifying data or performing incorrect actions. These perturbations are often so subtle that they are imperceptible to humans but highly effective against AI systems.

Cross-Medium Vulnerabilities

Adversarial attacks are robust and can manifest across various formats:

  • Digital Images: A small perturbation added to an image of a panda can cause a model to recognize it as a gibbon with high confidence.
  • Physical World: Adversarial examples can be printed on standard paper and photographed with a smartphone, still successfully fooling classifiers (e.g., labeling a "washer" as a "safe").
  • Critical Infrastructure: Attackers could potentially use stickers or paint to modify stop signs, causing autonomous vehicles to interpret them as "yield" signs or other markers.

Impact on Reinforcement Learning (RL)

Beyond image classification, reinforcement learning agents are also susceptible. Research involving DQN, TRPO, and A3C algorithms shows that adversarial inputs can degrade performance, causing an agent to move a pong paddle in the wrong direction or fail to spot enemies in Seaquest, even when perturbations are too subtle for human perception.

Attempted Defenses and Their Limitations

Traditional robustness techniques like weight decay and dropout are generally ineffective against adversarial examples. Only two specialized methods have provided significant defense, though both remain vulnerable to increased computational power from attackers.

Effective Defense Strategies

  • Adversarial Training: A brute-force approach where the model is explicitly trained on a large volume of generated adversarial examples to prevent it from being fooled by them.
  • Defensive Distillation: A strategy where a model is trained to output probabilities of different classes rather than hard decisions. These probabilities are supplied by a previous model trained on the same task, which smooths the model's surface and makes it harder for attackers to find exploitable tweaks.

The Failure of Gradient Masking

"Gradient masking" describes a category of failed defenses that attempt to deny an attacker access to a useful gradient. Since most attacks use the model's gradient to determine which direction to perturb an input to increase the probability of a wrong class, masking the gradient (e.g., by outputting only the most likely class instead of probabilities) seems effective.

However, gradient masking fails because:

  1. Persistence of Vulnerabilities: The "holes" in the model's defense still exist; the attacker simply has fewer clues to find them.
  2. Substitute Models: An attacker can train a "substitute model"—a smooth copy that imitates the defended model by observing its labels. The attacker then uses the substitute model's gradients to create adversarial examples that often successfully fool the original defended model.

OpenAI notes that both adversarial training and defensive distillation accidentally perform a form of gradient masking, often flattening the model rather than ensuring more points are classified correctly.

Why Defending Against Adversarial Examples is Difficult

Securing ML models against adversarial attacks is a complex research problem for several fundamental reasons:

  • Lack of Theoretical Models: Adversarial examples are solutions to non-linear and non-convex optimization problems. Because there are few theoretical tools to describe these solutions, it is difficult to prove that a defense will rule out a specific set of adversarial examples.
  • Input Space Complexity: Models must produce correct outputs for every possible input, yet most currently only work well on a very small fraction of all possible inputs they might encounter.
  • Non-Adaptive Defenses: Most current strategies are not adaptive; they may block one specific attack but leave other vulnerabilities open to an attacker who is aware of the defense mechanism.

Sources