OpenAI Research: Computational Limitations in Robust Classification

TL;DR

OpenAI has demonstrated that there are classification tasks where a robust classifier exists but is computationally impossible to learn. This discovery establishes a link between the hardness of robust classification and the existence of cryptographic primitives, creating a "win-win" scenario where either robust classifiers can be learned or new cryptographic primitives can be constructed.

Computational Hardness in Robust Classification

Robust classification refers to the learning of a classifier that remains accurate despite small perturbations to the input. Following the previous work of Bubeck, Lee, Price, and Razenshteyn, OpenAI researchers have extended the study of statistical and computational tradeoffs. They have identified specific classification tasks where an efficient robust classifier exists in the small-perturbation regime, and a non-robust classifier can be learned efficiently, but learning a robust classifier remains computationally hard assuming the hardness of factoring large numbers.

Impossible Robust Classification via Average-Case Hard Functions

OpenAI has demonstrated classification tasks where computationally efficient robust classification is impossible, even in cases where computationally unbounded robust classifiers exist. The researchers rely on the existence of average-case hard functions to prove that no efficient algorithm can learn a robust classifier for these specific tasks.

Hardness in the Large-Perturbation Regime

The research identifies tasks that are hard to robustly learn even when perturbations are large. In these scenarios, an efficient classifier that is robust to large perturbations exists, but it is computationally hard to learn any non-trivial robust classifier.

OpenAI utilizes two primary constructions to prove this hardness:

1. One-Way Functions: The first construction relies on the existence of one-way functions.
2. Learning Parity with Noise: The second construction relies on the hardness of the "learning parity with noise" problem. In this specific setting, an efficient algorithm exists that can generate fresh new labeled samples from polynomially many training examples, yet a robust classifier cannot be learned efficiently.

The "Win-Win" Scenario for Cryptography

The research concludes that any counterexample to efficient robust classification implies the existence of cryptographic primitives, such as one-way functions. This creates a a theoretical "win-win" scenario: either researchers can find a way to learn efficient robust classifiers for all such tasks, or the existence of these hard-to-learn tasks proves the existence of cryptographic primitives.