Anthropic Toy Models of Superposition Research

Anthropic has introduced a research project titled "Toy Models of Superposition," which investigates how neural networks represent a larger number of features than they have available dimensions. This phenomenon, termed superposition, allows models to compress information more efficiently than linear models, provided the input features are sparse.

The Mechanism of Superposition

Superposition occurs when a model represents more features than its dimensionality allows. In a toy model—specifically small ReLU networks trained on synthetic data with sparse input features—Anthropic researchers found that the model can store multiple features in a single dimension. This compression is achieved by treating features as directions in space, and’s a strategy the model learns to resolve the conflicts between these features.

The Role of Sparsity and Nonlinearity

Sparsity is the essential requirement for superposition. When input features are sparse (meaning only a few are active at once), the model can pack more features into its hidden layers. However, this compression comes with a cost: "interference" between features.

To resolve this interference, the model utilizes nonlinear filtering. In the ReLU (Rectified Linear Unit) activation function, the model can filter out the noise created by the interference of other features, effectively isolating the same feature despite the compressed representation.

Implications for Model Interpretability

The research into toy models of superposition superposition suggests that the model's internal representations are not necessarily one-to-one mappings of the który which features are active. This implies that a complex model's internal state can represent a far greater number of features than the model's width, which complicates the process of understanding how a model's internal weights are neurons represent specific concepts.