Why self-attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries

Chao Ma, Lexing Ying

In this paper, we show that structures similar to self-attention are natural to learn many sequence-to-sequence problems from the perspective of symmetry. Inspired by language proc…