23 citations · 24 across the 3 of their papers we have counts for
4 papers
What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
Julia Barth, Armin Sommer, Kunvar Thaman +1
Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but pursues a proxy that coincided with the intended goal during training…
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
Kunvar Thaman
Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems. We introduce the…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1168
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…
Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdouts
Jacob Haimes, Cenny Wenner, Kunvar Thaman +4
The training data for many Large Language Models (LLMs) is contaminated with test data. This means that public benchmarks used to assess LLMs are compromised, suggesting a performa…