Convergence and Divergence of Language Models under Different Random Seeds

Finlay Fehlauer, Kyle Mahowald, Tiago Pimentel

In this paper, we investigate the convergence of language models (LMs) trained under different random seeds, measuring convergence as the expected per-token Kullback--Leibler (KL)…