A phase transition between positional and semantic learning in a solvable model of dot-product attention

Hugo Cui, Freya Behrens, Florent Krzakala +2

Many empirical studies have provided evidence for the emergence of algorithmic mechanisms (abilities) in the learning of language models, that lead to qualitative improvements of t…