Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain

Hyowon Wi, Jeongwhan Choi, Noseong Park

Transformers have demonstrated remarkable performance across diverse domains. The key component of Transformers is self-attention, which learns the relationship between any two tok…