ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon

The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths. Existing efficient attention methods usually reduce t…