Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence

Itay Lavie, Kirsten Fischer, Andrey Lekov +3

Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present…