You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet

Zhen Qin, Yuxin Mao, Xuyang Shen +4

Linear attention mechanisms have gained prominence in causal language models due to their linear computational complexity and enhanced speed. However, the inherent decay mechanism…