9 papers
Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity
Zifan Lyu, Chahine Nejma, Tobias Wegel +2
Large Language Models are typically benchmarked by evaluating every model on every test query. For practitioners seeking the best model to deploy, this is often wasteful: if a mode…
Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead
Tom Sühr, Florian E. Dorner, Olawale Salaudeen +2
Large Language Models (LLMs) have achieved remarkable results on a range of standardized tests originally designed to assess human cognitive and psychological traits, such as intel…
Incentivizing Honesty among Competitors in Collaborative Learning and Optimization
Florian E. Dorner, Nikola Konstantinov, Georgi Pashaliev +1
Collaborative learning techniques have the potential to enable training machine learning models that are superior to models trained on a single entity's data. However, in many case…
Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget
Florian E. Dorner, Moritz Hardt
We study how to best spend a budget of noisy labels to compare the accuracy of two binary classifiers. It's common practice to collect and aggregate multiple noisy labels for a giv…
Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
Kazuki Egashira, Mark Vero, Jasper Dekoninck +3
Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs). While RLVR is designe…
Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data
Florian E. Dorner, Vivian Y. Nastl, Moritz Hardt
High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem. Scalable evaluation methods that avoid costly annotation have therefor…