OpenAI Testing Robustness Against Unforeseen Adversaries

OpenAI has developed a method and a new metric, Unforeseen Attack Robustness (UAR), to assess whether neural network classifiers can reliably defend against adversarial attacks they have not seen during training. This is critical because standard neural networks remain susceptible to adversarial examples—small, carefully crafted distortions that can cause a model to confidently misclassify an image while it remains recognizable to humans.

The Unforeseen Attack Robustness (UAR) Metric

The UAR metric evaluates the adversarial robustness of a single model against an unanticipated attack by comparing its performance to a strong defense that has prior knowledge of the distortion type. A UAR score near 100 indicates that a model's performance against an unforeseen attack is comparable to a defense specifically trained to resist that attack, representing a highly challenging objective for AI robustness.

Methodology for Assessing Robustness

OpenAI utilizes a three-step process to determine how a model performs against held-out distortion types:

1. Evaluation Against Diverse Distortion Types

Evaluating models solely against $L_{\infty}$ or $L_{2}$ distortion types is insufficient for predicting robustness against other types of attacks. OpenAI suggests that researchers evaluate models against adversarial distortions that differ significantly from those used during training. Recommended starting points for diverse attacks include:

  • $L_{1}$ distortions
  • $L_{2}$-JPEG distortions
  • Elastic distortions (flows along a local vector field)
  • Fog distortions (bounded magnitude fog-like distortions)
  • Gabor distortions (additive noise used to texture the image)
  • Snow distortions (adversarially constructed snowflakes)

2. Calibration of Distortion Sizes

Using too narrow a range of distortion sizes can lead to incorrect qualitative conclusions about robustness. OpenAI recommends choosing a calibrated range of distortion sizes based on the following criteria:

  • Human Recognizability: The range should be the largest possible range where images remain recognizable to humans.
  • Defense-Based Calibration: Because strong defenses force attackers to use larger distortion budgets to succeed, distortion sizes should be calibrated by evaluating against adversarially trained models.

3. Benchmarking Against Adversarially Trained Models

Robustness is benchmarked by comparing the model under test against models that have undergone adversarial training—a process where the model is trained on images that have already been attacked using a known distortion type.

Key Findings on Robustness Transfer

OpenAI's research indicates that adversarial robustness does not transfer broadly across different distortion types. Specifically:

  • Limited Transferability: Robustness gained through adversarial training against one distortion type does not necessarily protect the model against unforeseen distortions.
  • Negative Transfer: In some cases, hardening a model against one distortion (e.g., $L_{\infty}$) can actually reduce its robustness against another distortion (e.g., $L_{1}$). This suggests that increasing defense against a known attack can potentially harm the model's ability to defend against unanticipated attacks.

Sources