The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility

David Pape, Jonathan Evertz, Lea Schönherr

Progress in LLMs is increasingly measured through standardized benchmarks, where state-of-the-art improvements are often separated by fractions of a percentage point. At the same t…