4 papers
Reasoning Models Know What's Important, and Encode It in Their Activations
Yaniv Nikankin, Martin Tutek, Tomer Ashuach +2
Language models often solve complex tasks by generating long reasoning chains, consisting of many steps with varying importance. While some steps are crucial for generating the fin…
EvilGenie: A Reward Hacking Benchmark
Jonathan Gabor, Jayson Lynch, Jonathan Rosenfeld
We introduce EvilGenie, a benchmark for reward hacking in programming settings. We source problems from LiveCodeBench and create an environment in which agents can easily reward ha…
Crashing Waves vs. Rising Tides: Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Tasks
Matthias Mertens, Adam Kuzee, Brittany S. Harris +6
We characterize AI automation as a continuum between crashing waves, in which capabilities jump abruptly across narrow task sets, and rising tides, in which capabilities improve co…
On the Origin of Algorithmic Progress in AI
Hans Gundlach, Alex Fogelson, Jayson Lynch +4
Algorithms have been estimated to increase AI training FLOP efficiency by a factor of 22,000 between 2012 and 2023 [Ho et al., 2024]. Running small-scale ablation experiments on ke…