Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning

Shirin Amiraslani, Xin Gao

Sequence modeling tasks can involve intrinsic higher-order dependencies, while standard self-attention assigns scores to token pairs and does not explicitly parameterize such inter…