Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning

Yaomengxi Han, Debarghya Ghoshdastidar

The success of transformers is attributed to several factors, including their capacity to express formal languages over variable-length inputs, and ability of the attention mechani…