OpenAI Deep Double Descent

OpenAI has demonstrated that the double descent phenomenon occurs across various deep learning architectures, including CNNs, ResNets, and transformers. This behavior manifests as a pattern where model performance first improves, then worsens, and finally improves again as model size, data size, or training time increases.

Model-Wise Double Descent

Test error does not follow a monotonic path as model size increases; instead, it exhibits a "double descent" pattern. As the number of parameters in a neural network grows, test error initially decreases, then increases to a peak, and finally undergoes a second descent once the model is large enough to fit the training set.

This peak in test error occurs at the "critical regime" or interpolation threshold, where the model is just barely large enough to fit the training data. Factors that shift this interpolation threshold—such as changes to the optimization algorithm, the number of training samples, or the amount of label noise—also shift the location of the test error peak. The phenomenon is most prominent when label noise is added to the dataset, which amplifies the peak and makes it easier to observe.

Sample-Wise Non-Monotonicity

Increasing the amount of training data can, in specific regimes, lead to worse test performance. While adding more samples generally shifts the overall error curve downward, it also shifts the interpolation threshold and the associated test error peak to the right, because larger models are required to fit larger datasets.

For intermediate model sizes, these two effects can combine to create a regime where training on more data actually hurts performance. In one observed case involving transformers trained on a language-translation task, training on 4.5x more samples resulted in worse test performance for certain model sizes.

Epoch-Wise Double Descent

Training a model for a longer duration can reverse overfitting. For a fixed model size, as training proceeds through optimization steps, test and train error may decrease, increase, and then decrease again. This is referred to as epoch-wise double descent.

Across model size and optimization steps, the peak of test error consistently appears when models are just barely able to fit the training set.

Theoretical Intuition and Open Questions

OpenAI suggests that at the interpolation threshold, there is effectively only one model that fits the training data. Forcing this single model to fit noisy or misspecified labels destroys its global structure, leaving no "good models" that can both interpolate the training set and perform well on the test set.

In the over-parameterized regime, however, many models can fit the training set. The implicit bias of stochastic gradient descent (SGD) is believed to lead the optimization process toward "good models" that generalize well, although the exact mechanism remains an open research question.

Careful regularization and early stopping are noted as common methods to avoid the double descent peak.