Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers

Krzysztof Marcin Choromanski, Shanda Li, Valerii Likhosherstov +7

We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs). These inc…