Nonlinear Computation in Deep Linear Networks
OpenAI has demonstrated that deep linear networks, when implemented using floating-point arithmetic, are not mathematically linear and can be trained to perform nonlinear computation. By exploiting the way computers represent numbers, these networks can solve non-trivial problems that would typically require explicit nonlinear activation functions.
Floating-Point Underflow as a Source of Nonlinearity
In theoretical mathematics, consecutive linear layers in a neural network are equivalent to a single linear layer. However, computers use finite-precision representations, specifically the IEEE float32 standard, which consists of a sign bit, an 8-bit exponent, and a 23-bit fraction.
Nonlinearity emerges around zero due to "underflow." The smallest normal non-zero number is denoted as min ($1.0..0 \times 2^{-126}$). Any number smaller than this representable value is mapped to zero. This creates a significant gap and approximation error around zero, causing fundamental mathematical rules to fail:
- Distributive Law Failure: $(a + b) \times c$ may not equal $a \times c + b \times c$. For example, if $a = 0.4 \times min$, $b = 0.5 \times min$, and $c = 1/min$, the first expression results in $0$ (because $a+b$ underflows to $0$), while the second results in $0.9$.
- Associative Law Failure: $(a + b) + c$ may not equal $(b + c) + a$. If $a = 2.5 \times min$, $b = -1.6 \times min$, and $c = 1 \times min$, the first expression results in $min$, while the second results in $2.5 \times min$.
This behavior is further amplified in frameworks like TensorFlow, which builds primitives with denormals disabled (ftz=true), meaning any non-matrix multiply operation near the scale of $1e-38$ has an implicit nonlinearity.
Training Nonlinear Linear Networks with Evolution Strategies
Because modern differentiation libraries rely on symbolic differentiation, they are "blind" to these microscopic nonlinearities. Consequently, backpropagation cannot be used to train a network to exploit underflow.
To overcome this, OpenAI used Evolution Strategies (ES) to estimate gradients without symbolic differentiation. By ensuring activations remained sufficiently small to stay within the nonlinear range of float32, ES was able to find parameters that utilized these floating-point artifacts as computational nonlinearities.
MNIST Performance Comparison
When tested on the MNIST dataset, the difference in performance between standard training and ES-driven exploitation of floating-point nonlinearity was significant:
| Training Method | Training Accuracy | Test Accuracy |
|---|---|---|
| Backpropagation | 94% | 92% |
| Evolution Strategies (ES) | >99% | 96.7% |
This performance boost occurs because the floating-point nonlinearities allow each layer to generate novel features that are nonlinear combinations of lower-level features, a capability normally impossible for linear networks.
Future Implications
OpenAI suggests that this capability could be extended to other architectures, such as recurrent neural networks, or applied to improve complex machine learning tasks including translation and language modeling.