Symmetry Breaking in Transformers for Efficient and Interpretable Training

Eva Silverstein, Daniel Kunin, Vasudev Shyam

The attention mechanism in its standard implementation contains extraneous rotational degrees of freedom that are carried through computation but do not affect model activations or…