OpenAI Scaling Laws for Neural Language Models
OpenAI has identified empirical scaling laws that govern the performance of neural language models. These laws demonstrate that cross-entropy loss scales as a power-law relative to three primary variables: the number of model parameters, the size of the training dataset, and the total amount of compute used for training.
Predictable Performance Scaling
Language model performance, measured by cross-entropy loss, improves predictably as model size, dataset size, and compute increase. These power-law trends remain consistent across more than seven orders of magnitude.
While these primary drivers of performance are critical, other architectural choices—such as the specific network width or depth—have minimal effects on performance within a wide range of configurations.
Resource Allocation and Compute Efficiency
Simple equations can be used to determine the optimal allocation of a fixed compute budget to maximize performance. This optimization process relies on two key dependencies:
- Overfitting: The relationship between overfitting and the size of the model relative to the dataset.
- Training Speed: The relationship between training speed and the size of the model.
Sample Efficiency in Large Models
Larger models are significantly more sample-efficient than smaller models. Because of this efficiency, the most compute-efficient way to train a model is to utilize very large models trained on a relatively modest amount of data. To achieve maximum efficiency, training should be stopped significantly before the model reaches full convergence.