DiJiang: Efficient Large Language Models through Compact Kernelization

Hanting Chen, Zhicheng Liu, Xutao Wang +2

In an effort to reduce the computational load of Transformers, research on linear attention has gained significant momentum. However, the improvement strategies for attention mecha…