Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention

Tīkun Ông, Georg Bökman

We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group $G$, then $G$ can only act by permuting hea…