Do Transformers Need Three Projections? Analysis of QKV Variants

Key Takeaway: Projection Sharing Reduces Memory Overhead

Transformers can maintain performance while significantly reducing memory requirements by sharing weight projections. Specifically, sharing the Key and Value projections (K=V) allows for a massive reduction in the KV cache—up to 96.9% when combined with Multi-Query Attention (MQA)—with only marginal degradation in model perplexity. This suggests that the standard three-projection (Query, Key, Value) architecture may be more complex than necessary for many tasks.

Systematic Evaluation of QKV Variants

Researchers evaluated three primary constraints on the weight projections used in the attention mechanism to determine if all three are strictly necessary:

  • Shared Key-Value (K=V): The Key and Value projections are identical.
  • Shared Query-Key (Q=K): The Query and Key projections are identical, resulting in a symmetric attention map.
  • Single Projection (Q=K=V): All three projections share the same weights.

To mitigate the symmetry issues caused by Q=K and Q=K=V, the study employed 2D positional encodings to introduce asymmetry into the attention mechanism.

Performance Across Domains

Experiments were conducted across synthetic tasks, vision tasks (MNIST, CIFAR, TinyImageNet, and anomaly detection), and language modeling. The results indicate that projection-shared Transformers perform on par with, or occasionally better than, standard QKV Transformers.

Language Modeling Results

In language modeling tests using 300M and 1.2B parameter models trained on 10B tokens, the K=V variant demonstrated significant efficiency gains:

  • KV Cache Reduction: K=V projection sharing achieves a 50% reduction in the KV cache.
  • Perplexity Impact: This reduction comes with a minimal 3.1% degradation in perplexity.

Synergy with Head Sharing (GQA/MQA)

Projection sharing is complementary to existing head-sharing techniques like Grouped-Query Attention (GQA) and Multi-Query Attention (MQA). Combining these methods leads to drastic memory savings suitable for edge deployment:

  • K=V + GQA-4: Results in an 87.5% reduction in the KV cache.
  • K=V + MQA: Results in a 96.9% reduction in the KV cache.

Theoretical Insights: Why K=V Works

The study concludes that K=V preserves quality because keys and values often occupy similar representational spaces. Furthermore, because attention typically operates in a low-rank regime, the redundancy in the standard QKV formulation is often underutilized. Conversely, the Q=K variant tends to fail because it breaks the essential directionality of the attention mechanism.

Community Critique and Technical Discussion

While the results are promising for on-device inference, the machine learning community has raised several points regarding the generalizability and methodology of the study:

Training Scale and Over-training

One critic noted that the 1.2B parameter model was trained on only 10B tokens, which is significantly below the Chinchilla-optimal compute budget. There is a concern that simplifications to attention may appear effective in under-trained regimes but fail to scale to the "over-trained" levels (trillions of tokens) seen in modern LLMs, where the full expressiveness of standard attention becomes more critical.

Mathematical Notation

Some readers pointed out a lack of clarity in the paper's notation, specifically the use of "Q-K=V" to denote shared Key-Value projections (K=V) rather than a mathematical subtraction operation.

Inductive Bias and Architecture

Discussion among practitioners suggests that the standard QKV formulation is widely used not necessarily because it is the optimal mathematical approach, but because it is computationally efficient on GPUs and provides a safe, well-understood baseline for scaling.

Sources