Early Weight Averaging meets High Learning Rates for LLM Pre-training

Sunny Sanyal, Atula Neerkaje, Jean Kaddour +2

Training Large Language Models (LLMs) incurs significant cost; hence, any strategy that accelerates model convergence is helpful. In this paper, we investigate the ability of a sim…