Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning

Joy Crosbie, Ekaterina Shutova

Large language models (LLMs) have shown a remarkable ability to learn and perform complex tasks through in-context learning (ICL). However, a comprehensive understanding of its int…