Transformer Normalisation Layers and the Independence of Semantic Subspaces

Stephen Menary, Samuel Kaski, Andre Freitas

Recent works have shown that transformers can solve contextual reasoning tasks by internally executing computational graphs called circuits. Circuits often use attention to logical…