Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling

Teodora Srećković, Jonas Geiping, Antonio Orvieto

Adam is known to perform significantly better than Stochastic Gradient Descent (SGD) in language models, a phenomenon for which a number of explanations have been proposed. In this…