Kimi Linear: An Expressive, Efficient Attention Architecture

Kimi Linear Outperforms Full Attention in Performance and Efficiency

Kimi Linear is a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across short-context, long-context, and reinforcement learning (RL) scaling regimes. By combining a new expressive linear attention module with Multi-Head Latent Attention (MLA), the architecture reduces KV cache usage by up to 75% and increases decoding throughput by up to 6 times for 1-million-token contexts.

Kimi Delta Attention (KDA): Enhancing RNN Memory

At the core of Kimi Linear is Kimi Delta Attention (KDA), an expressive linear attention module designed to optimize the use of limited finite-state RNN memory. KDA extends Gated DeltaNet by implementing a finer-grained gating mechanism, which allows the model to more effectively manage and update its internal state.

To ensure high hardware efficiency, the Kimi team developed a bespoke chunkwise algorithm. This algorithm utilizes a specialized variant of Diagonal-Plus-Low-Rank (DPLR) transition matrices. This specific implementation reduces the computational overhead associated with general DPLR formulations while remaining consistent with the classical delta rule, enabling faster processing without sacrificing expressiveness.

Hybrid Architecture and Scaling Results

Kimi Linear employs a layerwise hybrid approach, integrating KDA with Multi-Head Latent Attention (MLA). The researchers pretrained a model with 3 billion activated parameters and 48 billion total parameters.

Using an identical training recipe to full MLA models, Kimi Linear demonstrated a sizeable margin of improvement across all evaluated tasks. The architecture is positioned as a drop-in replacement for full attention architectures, specifically excelling in tasks requiring longer input and output lengths.

Ecosystem and Implementation

To facilitate further research and adoption, the Kimi team has open-sourced the following components:

• The KDA kernel
• vLLM implementations
• Pre-trained and instruction-tuned model checkpoints

Community Insights and Evolution

Industry practitioners and researchers have noted the architectural evolution of Kimi Linear and its relationship to other models. Specifically, the Kimi K3 model is heavily based on the Kimi Linear architecture, scaling it up and incorporating native vision capabilities and RL improvements.

Other developers have observed that Gated DeltaNet 2 represents a further evolution in expressiveness over the original Kimi Linear approach. Additionally, some community members have questioned how these linear attention hybrids perform on specific long-context retrieval tasks, such as "needle in a haystack" tests, which have historically been a point of failure for linear attention models.

"If you read the recently-released Kimi K3 paper, you'll see that it's heavily based on Kimi Linear discussed here, scaling it up and adding a bunch more things (like native vision and RL improvements)."