On the Role of Attention Masks and LayerNorm in Transformers

Xinyi Wu, Amir Ajorlou, Yifei Wang +2

Self-attention is the key mechanism of transformers, which are the essential building blocks of modern foundation models. Recent studies have shown that pure self-attention suffers…