14 citations · 17 across the 4 of their papers we have counts for
4 papers
Attention Residuals
Kimi Team, Guangyu Chen, Yu Zhang +34
Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-st…
Kimi Linear: An Expressive, Efficient Attention Architecture
Kimi Team, Yu Zhang, Zongyu Lin +57
We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including s…
Kimi K2: Open Agentic Intelligence
Kimi Team, Yifan Bai, Yiping Bao +195
We introduce Kimi K2, a Mixture-of-Experts (MoE) large language model with 32 billion activated parameters and 1 trillion total parameters. We propose the MuonClip optimizer, which…
CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models
Guanduo Chen, Yutong He, Yipeng Hu +2
Large Language Models (LLMs) demonstrate exceptional performance across various tasks but demand substantial computational resources even for fine-tuning computation. Although Low-…