Predicting the Emergence of Induction Heads in Language Model Pretraining

Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan Schneider

Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise char…