OpenAI AI and Compute Analysis

Exponential Growth in AI Compute

Since 2012, the amount of compute used in the largest AI training runs has increased exponentially, doubling every 3.4 months. This growth rate significantly exceeds Moore's Law, which historically had a doubling period of approximately two years. Over the period since 2012, this metric has grown by more than 300,000x, whereas a Moore's Law doubling period would have resulted in only a 7x increase.

OpenAI notes that improvements in compute have been a key component of AI progress, suggesting that the industry should prepare for the implications of systems with capabilities far beyond those currently available.

Historical Eras of AI Compute

Analysis of data spanning from 1959 to 2012 reveals two distinct eras of AI training compute usage:

The First Era (1959–2012): AI results roughly tracked the growth of Moore's Law.
The Modern Era (2012–Present): AI training compute has substantially outpaced macro trends in hardware growth.

Notably, the trend of compute usage has remained consistent regardless of the"AI winters" or periods of excitement, suggesting that these cycles of investment and hype have had little effect on the actual computational power used to train learning systems over the last half-century.

Methodology for Estimating Compute

OpenAI utilized two primary methodologies to estimate the total compute used to train well-known AI results, measured in petaflop/s-days (pfs-days).

Method 1: Counting Operations

This method involves directly counting the number of floating-point operations (FLOPs)—specifically adds and multiplies—in the described architecture per training example, then multiplying by the total number of forward and backward passes during training. This approach is used when the architecture and training details are fully disclosed.

Method 2: GPU Training Time

When direct operation counts are unavailable, OpenAI estimates compute based on the number of GPUs used, the duration of training, and an assumed utilization efficiency. The standard assumptions are:

GPU Utilization: 33% (0.33)
CPU Utilization: 17% (0.17)

OpenAI states that these calculations are not intended to be precise but aim to be correct within a factor of 2–3, focusing on the order of magnitude.

Compute Requirements for Notable AI Results

While massive compute is not a requirement for every important breakthrough, OpenAI identifies several noteworthy results that were achieved using modest amounts of compute:

Attention is all you need (Transformer): 0.089 pfs-days
Adam Optimizer: < 0.0007 pfs-days
Learning to Align and Translate: 0.018 pfs-days
GANs: < 0.006 pfs-days
Word2Vec: < 0.00045 pfs-days
Variational Auto Encoders: < 0.0000055 pfs-days