Understanding Transformer Architecture as Associative Memory

Modern transformer architectures, including MLPs and self-attention layers, can be intuitively understood as associative memory stores. By viewing these layers as mechanisms for storing and retrieving key-value pairs, we can derive their capacity limits based on the interference-to-signal ratio—a metric that quantifies how much stored memories obstruct the retrieval of a target fact.

The Linear Memory Store

The most fundamental memory store is a linear layer, which accumulates memories using outer products: $W = \sum_{i=1}^{M} v_i k_i^T$. In this model, a query $x$ retrieves a value by calculating the similarity between the query and all stored keys, returning a sum of values weighted by these similarities.

However, this approach suffers from interference. When we store $M$ memories, the recall for a target key $k_j$ is $f(k_j) = v_j + \sum_{i\neq j} v_i(k_i^T k_j)$. As $M$ grows, the interference terms accumulate. For random keys, the capacity—the point where interference matches the signal—is approximately $M \approx d_k$, where $d_k$ is the key dimension.

Improving Capacity with Optimal Linear Stores and MLPs

We can improve capacity by refining how we write and read from memory. The optimal linear store uses a least-squares approach to "whiten" the keys, effectively canceling interference and doubling capacity to $M \approx 2d_k$.

MLPs introduce non-linearities that further suppress interference. By passing similarities through a function $\sigma$, such as ReLU or squared ReLU, the model suppresses low-similarity keys that would otherwise contribute noise.

• ReLU MLPs: Zeroing negative similarities reduces interference variance, pushing capacity to $M \approx 2d_k$.
• Squared ReLU (GLU/SwiGLU): Further suppressing low-similarity keys allows for significantly higher capacity, scaling with $d_k^2$.

These findings align with research suggesting that feedforward layers in transformers function as key-value stores where neurons activate based on recognizable input patterns.

Self-Attention as Non-Parametric Memory

Self-attention represents the limit of this approach by using the softmax function to perform a weighted retrieval. Unlike MLPs, where keys and values are fixed parameters, self-attention computes these from the input tokens, making it a non-parametric store where capacity scales with context length.

By using the exponential function inherent in softmax, self-attention can achieve massive capacity. When scaled with $\beta = \sqrt{d_k}$, the capacity grows exponentially rather than polynomially, reaching over a million memories in standard configurations. To maintain this performance in long-context scenarios, models often scale the attention logits using $\log M$ heuristics (such as those found in Qwen or YaRN), which keeps the interference-to-signal ratio stable as the context grows.

Insights and Discussion

This framework provides a bridge between communications engineering and deep learning. As noted in community discussions, the "interference-to-signal" perspective is highly familiar to electrical engineers, suggesting that techniques for maximizing channel capacity are directly applicable to understanding how neural networks store information.

While the mathematical abstraction is powerful, some practitioners emphasize that understanding these systems requires concrete examples. In an LLM context, the "keys" are the tokens in the context window, and the "values" are the corresponding hidden state representations that the model uses to predict subsequent tokens. As one commenter noted, the efficiency of transformers compared to traditional Hopfield networks lies in their ability to retrieve memories in a single forward pass, rather than requiring iterative convergence.

Ultimately, viewing transformers as memory stores highlights that intelligence in these models is deeply tied to their ability to compress and retrieve vast amounts of information without catastrophic interference.