1 citations · 2 across the 11 of their papers we have counts for
4 papers · 1 filter
Kimi Linear: An Expressive, Efficient Attention Architecture
Kimi Team, Yu Zhang, Zongyu Lin +57
We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including s…
WuNeng: Hybrid State with Attention
Liu Xiao, Li Zhiyuan, Lin Yueyu
The WuNeng architecture introduces a novel approach to enhancing the expressivity and power of large language models by integrating recurrent neural network (RNN)-based RWKV-7 with…
State Tuning: State-based Test-Time Scaling on RWKV-7
Liu Xiao, Li Zhiyuan, Lin Yueyu
Test-time scaling has emerged as a prominent research direction in machine learning, enabling models to enhance their expressive capabilities during inference.Transformers, renowne…
ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer
Lin Yueyu, Li Zhiyuan, Peter Yue +1
As is known, hybrid quadratic and subquadratic attention models in multi-head architectures have surpassed both Transformer and Linear RNN models , with these works primarily focus…