Kimi Linear: An Expressive, Efficient Attention Architecture
Authors
Kimi Team
Yu Zhang
Zongyu Lin
Xingcheng Yao
Jiaxi Hu
Fanqing Meng
Chengyin Liu
Xin Men
Songlin Yang
Zhiyuan Li
Wentao Li
Enzhe Lu
Weizhou Liu
Yanru Chen
Weixin Xu
Longhui Yu
Yejie Wang
Yu Fan
Longguang Zhong
Enming Yuan
Dehao Zhang
Yizhi Zhang
T. Y. Liu
Haiming Wang
Shengjun Fang
Weiran He
Shaowei Liu
Yiwei Li
Jianlin Su
Jiezhong Qiu
Bo Pang
Junjie Yan
Zhejun Jiang
Weixiao Huang
Bohong Yin
Jiacheng You
Chu Wei
Zhengtao Wang
Chao Hong
Yutian Chen
Guanduo Chen
Yucheng Wang
Huabin Zheng
Feng Wang
Yibo Liu
Mengnan Dong
Zheng Zhang
Siyuan Pan
Wenhao Wu
Yuhao Wu
Longyu Guan
Jiawen Tao
Guohong Fu
Xinran Xu
Yuzhi Wang
Guokun Lai
Yuxin Wu
Xinyu Zhou
Zhilin Yang
Yulun Du
Abstract
We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule.
We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths.
To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.
