Intrinsic and Extrinsic Organized Attention: Softmax Invariance and Network Sparsity

Oluwadamilola Fasina, Ruben V. C. Pohle, Pei-Chun Su +1

We examine the intrinsic (within the attention head) and extrinsic (amongst the attention heads) structure of the self-attention mechanism in transformers. Theoretical evidence for…