Output Embedding Centering for Stable LLM Pretraining

Felix Stollenwerk, Anna Lokrantz, Niclas Hertzberg

Pretraining of large language models is not only expensive but also prone to certain training instabilities. A specific instability that often occurs at the end of training is outp…