In-Context Learning Without Copying

Kerem Sahin, Sheridan Feucht, Adam Belfki +4

Induction heads are attention heads that perform inductive copying by matching patterns from earlier context and copying their continuations verbatim. As models develop induction h…