Sampling at intermediate temperatures is optimal for training large language models in protein structure prediction

L. Ghiringhelli, A. Zambon, G. Tiana

We investigate the parameter space of transformer models trained on protein sequence data using a statistical mechanics framework, sampling the loss landscape at varying temperatur…