3 papers
cs.LG2026
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
Tiberiu Musat, Tiago Pimentel, Nicolas Zucchet +1
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics h…
cs.AI2025
On the Emergence of Induction Heads for In-Context Learning
Tiberiu Musat, Tiago Pimentel, Lorenzo Noci +3
Transformers have become the dominant architecture for natural language processing. Part of their success is owed to a remarkable capability known as in-context learning (ICL): the…
cs.LG2024
Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
Tiberiu Musat
In this paper, I introduce the retrieval problem, a simple yet common reasoning task that can be solved only by transformers with a minimum number of layers, which grows logarithmi…