FNetAR: Mixing Tokens with Autoregressive Fourier Transforms

Tim Lou, Michael Park, Mohammad Ramezanali +1

In this note we examine the autoregressive generalization of the FNet algorithm, in which self-attention layers from the standard Transformer architecture are substituted with a tr…