Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models

Kwok Chun Au, Adam Block

Many modern Language Model (LM) pipelines return an averaged model, such as an exponential moving average of the training iterates, rather than the final iterate itself. This raise…