Local to Global: Learning Dynamics and Effect of Initialization for Transformers

Ashok Vardhan Makkuva, Marco Bondaschi, Chanakya Ekbote +4

In recent years, transformer-based models have revolutionized deep learning, particularly in sequence modeling. To better understand this phenomenon, there is a growing interest in…