What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation

Aaditya K. Singh, Ted Moskovitz, Felix Hill +2

In-context learning is a powerful emergent ability in transformer models. Prior work in mechanistic interpretability has identified a circuit element that may be critical for in-co…