Anthropic Reveals Privileged Bases in Transformer Residual Streams

TL;DR

Anthropic found that individual coordinates in a transformer's residual stream are not mathematically arbitrary; they align with privileged bases, a phenomenon they attribute primarily to Adam's per‑dimension normalizers rather than layer normalization or floating‑point precision.

Key Finding: Basis Alignment Exists

Empirical analysis shows that residual‑stream dimensions carry systematic information, contradicting the theoretical expectation that basis directions should be interchangeable and carry no special meaning.

Investigated Causes

Adam Optimizer Normalizers

The authors provisionally identify the per‑dimension normalizers in the Adam optimizer as the most plausible source of the observed basis alignment. Adam scales each parameter individually, which can introduce consistent directional biases in the learned representations.

Layer Normalization Ruled Out

Experiments demonstrate that layer normalization does not produce the privileged‑basis effect. Changing or removing layer‑norm layers leaves the alignment unchanged, indicating it is not the cause.

Finite‑Precision Arithmetic Ruled Out

Tests varying floating‑point precision (e.g., float32 vs. float64) show no impact on basis alignment, allowing the authors to confidently dismiss finite‑precision calculations as the source.

Implications for Transformer Theory

The discovery challenges the assumption that the residual stream's coordinate system is arbitrary. If optimizer dynamics imprint structure onto the basis, theoretical models of transformers must account for optimizer‑induced anisotropy when analyzing representation geometry.

Broader Context

Anthropic situates this work alongside other frontier‑model investigations, such as:

• Emerging Multi‑Agent Systems – exploring behavioral patterns that can cause systemic failures.
• Worker Retraining Evidence Review – a policy‑oriented review co‑authored with external researchers.
• Claude’s Mathematical Capabilities – highlighting a research version of Claude that advanced a bound related to the Riemann hypothesis.

These links illustrate Anthropic’s broader focus on both technical and societal aspects of advanced AI.

Conclusion

The privileged‑basis phenomenon appears tied to Adam's per‑dimension scaling, not to layer normalization or numeric precision. Recognizing optimizer‑driven basis bias is essential for accurate theoretical modeling of transformer internals and may influence future architecture and training‑algorithm design.