Kimi Linear: An Expressive, Efficient Attention Architecture

Kimi Linear Outperforms Full Attention in Efficiency and Performance

Kimi Linear is a hybrid linear attention architecture that achieves superior performance compared to full attention across short-context, long-context, and reinforcement learning (RL) scaling regimes. By replacing standard attention with a more efficient mechanism, the architecture reduces KV cache usage by up to 75% and increases decoding throughput by up to 6 times for 1 million token contexts.

Kimi Delta Attention (KDA): The Core Innovation

At the center of Kimi Linear is Kimi Delta Attention (KDA), an expressive linear attention module designed to optimize how models use limited finite-state RNN memory.

Finer-Grained Gating

KDA extends the Gated DeltaNet architecture by implementing a finer-grained gating mechanism. This allows the model to more effectively manage and update its internal state, increasing the expressiveness of the linear attention process.

Hardware-Efficient Chunkwise Algorithm

To ensure high hardware efficiency, Kimi Linear utilizes a bespoke chunkwise algorithm. This algorithm employs a specialized variant of Diagonal-Plus-Low-Rank (DPLR) transition matrices. This specific implementation reduces the computational overhead associated with general DPLR formulations while remaining consistent with the classical delta rule, enabling faster processing without sacrificing mathematical rigor.

Hybrid Architecture and Scaling Results

Kimi Linear employs a layerwise hybrid approach, combining KDA with Multi-Head Latent Attention (MLA).

Model Specifications

The researchers pretrained a model with 3 billion activated parameters and 48 billion total parameters. Using an identical training recipe to full MLA models, the Kimi Linear model demonstrated a sizeable performance margin over full MLA across all evaluated tasks.

Efficiency Gains

For contexts reaching 1 million tokens, the architecture provides significant operational advantages:

• KV Cache Reduction: Up to 75% less memory usage compared to full attention.
• Throughput Increase: Up to 6x increase in decoding throughput.

Ecosystem and Research Impact

To facilitate further research and adoption, the Kimi team has open-sourced the KDA kernel, vLLM implementations, and both pre-trained and instruction-tuned model checkpoints.

Integration with Future Models

Community discussion indicates that Kimi Linear serves as the foundation for subsequent releases. Specifically, the Kimi K3 paper highlights that the K3 model is heavily based on Kimi Linear, scaling the architecture further and integrating native vision capabilities and RL improvements.

Community Perspectives

While the technical results are promising, some practitioners have noted the evolution of the field, mentioning that Gated DeltaNet 2 appears to be a further evolution in expressiveness. Other users have raised questions regarding the performance of linear attention hybrids on specific long-context retrieval tasks, such as "needle in a haystack" tests, where such architectures have historically struggled compared to full attention.