4 papers
Active Model Selection for Large Language Models
Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2
We introduce LLM SELECTOR, the first framework for active model selection of Large Language Models (LLMs). Unlike prior evaluation and benchmarking approaches that rely on fully an…
Evaluating the Robustness of Chinchilla Compute-Optimal Scaling
Rylan Schaeffer, Noam Levi, Andreas Kirsch +4
Hoffman et al (2022)'s Chinchilla paper introduced the principle of compute-optimal scaling, laying a foundation for future scaling of language models. In the years since, however,…
When three experiments are better than two: Avoiding intractable correlated aleatoric uncertainty by leveraging a novel bias--variance tradeoff
Paul Scherer, Andreas Kirsch, Jake P. Taylor-King
Real-world experimental scenarios are characterized by the presence of heteroskedastic aleatoric uncertainty, and this uncertainty can be correlated in batched settings. The bias--…
(Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Models
Andreas Kirsch
Epistemic uncertainty is crucial for safety-critical applications and data acquisition tasks. Yet, we find an important phenomenon in deep learning models: an epistemic uncertainty…