Spatial Induction Heads: In-Context Learning of Multidimensional Cellular Automata

Kimia Kazemian, Menghan Xu, John Thickstun +1

Induction heads provide a mechanistic account of in-context learning in sequential data, but existing theory largely assumes that the context relevant to a prediction forms a conti…