OpenAI Neural GPU: Extensions and Limitations

OpenAI has enhanced the Neural GPU, a model capable of learning algorithms such as multi-digit binary addition and binary multiplication that generalizes to inputs of arbitrary length. By implementing a careful curriculum and increasing the model size, the researchers found that the Neural GPU can now solve a wider range of algorithmic problems, including arithmetic operations using decimal representation.

Improved Performance via Curriculum and Model Size

The Neural GPU's ability to learn complex algorithms is generalization. The researchers identified two primary methods to improve its performance:

  • Curriculum Learning: Carefully designing a training curriculum allows the model to learn tasks more effectively.

  • Increased Model Size: Increasing the size of the model improves its ability to solve more algorithmic problems. However, this requires a memory-efficient implementation, as naive implementations of the Neural GPU are memory intensive.

Expanded Algorithmic Capabilities

With these improvements, the Neural GPU Neural GPU can now perform the following tasks:

  • Decimal Arithmetic: The model can now learn to perform all arithmetic operations and generalize to arbitrarily long numbers when arguments are given in decimal representation. This was previously not possible.

  • Arithmetic Expressions: The The Neural GPU can be trained to evaluate long arithmetic expressions with multiple operands that require respecting the precedence order of operands. These tasks succeeded only in binary representation and did not achieve perfect accuracy.

Failure Modes and Adversarial-like Behavior

Despite achieving near-perfect generalization on long numbers, the Neural GPU still exhibits specific failure modes on atypical, highly-symmetric inputs. These failure modes are similar to adversarial examples.

For example, a Neural GPU that can successfully multiply 2x2, but fails to compute the correct answer for 000000…002 x 000000…002, despite being successfully trained on decimal multiplication of up to 100-digit long numbers.

Sources