Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation

Yingyi Chen, Qinghua Tao, Francesco Tonin +1

Recently, a new line of works has emerged to understand and improve self-attention in Transformers by treating it as a kernel machine. However, existing works apply the methods for…