12 papers
Using predictive multiplicity to measure individual performance within the AI Act
Karolin Frohnapfel, Mara Seyfert, Sebastian Bordt +2
When building AI systems for decision support, one often encounters the phenomenon of predictive multiplicity: a single best model does not exist; instead, one can construct many m…
We Need Explanation Cards to Connect Explanation Algorithms to the Real World
Eric Günther, Balázs Szabados, Kristof Meding +3
Algorithmic explanations are intended to help stakeholders understand opaque algorithmic decisions, but in practice, they often fall short. First, the meaning of algorithmic explan…
Validity Threats for Foundation Model Research
Gunnar König, Martin Pawelczyk, Ulrike von Luxburg +1
Controlled experiments are the backbone of machine learning research, but at the scale of modern foundation models, they have become prohibitively expensive. Instead, the community…
Weight Decay Improves Language Model Plasticity
Tessa Han, Sebastian Bordt, Hanlin Zhang +1
Large language models are typically trained in two broad phases: pretraining to produce a base model, followed by further training to improve downstream performance. However, hyper…
How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
Leena Chennuru Vankadara, Moritz Haas, Luke Hayward +2
Recent frontier large language models predominantly rely on Mixture-of-Experts (MoE) architectures. Despite empirical progress, there is still no principled understanding of how hy…
Train Once, Answer All: Many Pretraining Experiments for the Cost of One
Sebastian Bordt, Martin Pawelczyk
Recent work has demonstrated that controlled pretraining experiments are a powerful tool for studying the relationship between training data and large language model (LLM) behavior…