Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling

Ryan Solgi, Parsa Madinei, Zheng Zhang

The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tenso…