Rethinking Associative Memory Mechanism in Induction Head

Shuo Wang, Issei Sato

Induction head mechanism is a part of the computational circuits for in-context learning (ICL) that enable large language models (LLMs) to adapt to new tasks without fine-tuning. M…