Attention Is Not What You Need: Grassmann Manifold Architectures
Core Concept: Replacing Self-Attention with Grassmann Flows
The paper "Attention Is Not What You Need" (arXiv:2512.19428) challenges the necessity of explicit self-attention in sequence modeling. The authors argue that standard multi-head attention functions as a form of "tensor lifting," where hidden vectors are mapped into a high-dimensional space of pairwise interactions. While expressive, this approach is mathematically opaque, making it difficult to define explicit invariants for model behavior.
To address this, the authors propose a Causal Grassmann layer. Instead of constructing an L-by-L attention matrix, this architecture:
- Linearly reduces token states.
- Encodes local token pairs as two-dimensional subspaces on a Grassmann manifold using Plucker coordinates.
- Fuses these geometric features back into hidden states via gated mixing.
By operating on a finite-dimensional manifold rather than an unstructured tensor space, the model propagates information through controlled deformations of low-rank subspaces over local windows. This approach achieves linear scaling in sequence length for a fixed rank.
Performance and Benchmarking
The proposed architecture was evaluated on two primary benchmarks to demonstrate its viability against standard Transformers:
- Wikitext-2: Purely Grassmann-based models with 13 to 18 million parameters achieved validation perplexities within 10 to 15 percent of size-matched Transformers.
- SNLI (Natural Language Inference): A Grassmann-Plucker head applied to DistilBERT achieved a validation accuracy of 0.8550 and a test accuracy of 0.8538, slightly outperforming the standard Transformer head (0.8545 and 0.8511, respectively).
Community Critique and Technical Considerations
The research has sparked debate within the technical community regarding the methodology, scale, and presentation of the work. Critics have raised several points of concern:
- Scale and Generalization: Commenters noted that the experiments are limited to small-scale datasets and models. Critics argue that performance on toy problems does not necessarily translate to the large-scale training regimes where Transformers currently dominate.
- Mathematical Trade-offs: Some observers pointed out that using Grassmannian manifolds involves a trade-off, as the model preserves subspace orientation while potentially losing information about the underlying linear map and basis.
- Implementation Details: A potential issue was identified regarding the pairing operation (zt, zt+Delta), with some suggesting that using future tokens to predict the next token could lead to data leakage if not handled correctly.
- Presentation and Rigor: Discussion on platforms like Hacker News highlighted a perceived lack of visual aids, such as schematics or graphs, which some readers felt were necessary to support the paper's claims. There were also questions regarding the lack of an established academic footprint for the author, leading some to scrutinize the document's origin and depth of analysis.
While the paper offers a novel geometric perspective on sequence modeling, the consensus among early reviewers is that further evidence is required to determine if manifold-based designs can compete with the scaling efficiency of modern Transformer architectures.