Scale-invariant Attention

Ben Anson, Xi Wang, Laurence Aitchison

One persistent challenge in LLM research is the development of attention mechanisms that are able to generalise from training on shorter contexts to inference on longer contexts. W…