PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs

Oskar van der Wal, Pietro Lesci, Max Muller-Eberstein +4

The stability of language model pre-training and its effects on downstream performance are still understudied. Prior work shows that the training process can yield significantly di…