3 papers
cs.LG2025
Pre-Training LLMs on a budget: A comparison of three optimizers
Joel Schlotthauer, Christian Kroos, Chris Hinze +3
Optimizers play a decisive role in reducing pre-training times for LLMs and achieving better-performing models. In this study, we compare three major variants: the de-facto standar…
cs.CL2025
From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models
Viktor Hangya, Fabian Küch, Darina Gold
Iterative evaluation of LLMs during training is essential to ensure expected capability development, but can be time- and compute-intensive. While NLU tasks, where the model select…
cs.CL2024
Data Processing for the OpenGPT-X Model Family
Nicolo' Brandizzi, Hammam Abdelwahab, Anirban Bhowmick +19
This paper presents a comprehensive overview of the data preparation pipeline developed for the OpenGPT-X project, a large-scale initiative aimed at creating open and high-performa…