Beyond Self Attention: A Subquadratic Fourier Wavelet Transformer with Multi Modal Fusion

Andrew Kiruluta, Andreas Lemos, Eric Lundy

We revisit the use of spectral techniques to replaces the attention mechanism in Transformers through Fourier Transform based token mixing, and present a comprehensive and novel re…