Translational Equivariance in Kernelizable Attention

Max Horn, Kumar Shridhar, Elrich Groenewald +1

While Transformer architectures have show remarkable success, they are bound to the computation of all pairwise interactions of input element and thus suffer from limited scalabili…