In-context Learning and Induction Heads

Catherine Olsson, Nelson Elhage, Neel Nanda +23

"Induction heads" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect ev…