The emergence of sparse attention: impact of data distribution and benefits of repetition

Nicolas Zucchet, Francesco d'Angelo, Andrew K. Lampinen +1

Emergence is a fascinating property of large language models and neural networks more broadly: as models scale and train for longer, they sometimes develop new abilities in sudden…