Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8

Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that…