Sampling at intermediate temperatures is optimal for training large language models in protein structure prediction

L. Ghiringhelli, A. Zambon, G. Tiana

Using a statistical mechanics framework, we investigate the parameter space of transformer models trained on protein sequence data. We sample the loss landscape at varying temperat…