Modes of Sequence Models and Learning Coefficients

Zhongtian Chen, Daniel Murfet

We develop a geometric account of sequence modelling that links patterns in the data to measurable properties of the loss landscape in transformer networks. First, we cast conditio…