Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4

We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling jointly optimal learning rates…