Pre-Training LLMs on a budget: A comparison of three optimizers

Joel Schlotthauer, Christian Kroos, Chris Hinze +3

Optimizers play a decisive role in reducing pre-training times for LLMs and achieving better-performing models. In this study, we compare three major variants: the de-facto standar…