Anthropic Research: Superposition, Memorization, and Double Descent

Anthropic researchers have identified that simple neural networks can use superposition to represent more features than they have available neurons, particularly when memorizing limited datasets. This discovery suggests that superposition is a central mechanism for how models overfit and memorize specific examples, which is critical for achieving mechanistic interpretability.

The Role of Superposition in Memorization

Superposition occurs when a neural network represents more features than its dimensionality (the number of neurons) allows. In the case of a language model memorizing text verbatim, the network may avoid the inefficient use of one neuron per memorized sequence. Instead, it leverages superposition to store these mutually exclusive cases in a single neuron or a set of neurons, preventing interference between the different memorized sequences.

Mechanistic Interpretability and Overfitting

Mechanistic interpretability aims to understand the internal workings of deep learning models. Anthropic's research indicates that there is a link between overfitting—a central problem in machine learning—and the learning of interpretable features. Because there currently is little mechanistic understanding of what happens when models overfit or memorize, studying these phenomena in toy models provides a necessary foundation for understanding how complex models behave during the training process.

Investigation of Toy Models

Anthropic conducted a preliminary investigation by training the same toy models used in their previous research on limited datasets. This study shifted the focus from the infinite-data, underfitting regime to the case of limited data, where overfitting occurs. Even in these simplified models, the research team found that the toy model serves as a rich case study for understanding the relationship between superposition and the model's ability to memorize data.