Controlling changes to attention logits

Ben Anson, Laurence Aitchison

Stability of neural network weights is critical when training transformer models. The query and key weights are particularly problematic, as they tend to grow large without any int…