3 citations · 7 across the 7 of their papers we have counts for
4 papers · 1 filter
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
Mohit Raghavendra, Soham Dan, Miguel Romero Calvo +12
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refac…
Imitation Learning for Multi-turn LM Agents via On-policy Expert Corrections
Niklas Lauffer, Xiang Deng, Srivatsa Kundurthy +2
A popular paradigm for training LM agents relies on imitation learning, fine-tuning on expert trajectories. However, we show that the off-policy nature of imitation learning for mu…
Remote Labor Index: Measuring AI Automation of Remote Work
Mantas Mazeika, Alice Gatti, Cristina Menghini +44
AIs have made rapid progress on research-oriented benchmarks of knowledge and reasoning, but it remains unclear how these gains translate into economic value and automation. To mea…
The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems
Richard Ren, Arunim Agarwal, Mantas Mazeika +13
As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting th…