Beyond Nyströmformer -- Approximation of self-attention by Spectral Shifting

Madhusudan Verma

Transformer is a powerful tool for many natural language tasks which is based on self-attention, a mechanism that encodes the dependence of other tokens on each specific token, but…