8 citations · 14 across the 38 of their papers we have counts for
20 papers · 1 filter
Into the Unknown: Benchmarking AI agents on open scientific research problems
Seungone Kim, Jiayi Geng, Young-Jun Lee +25
Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper along with a research direction…
EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights
Jiayi Geng, Zhengxuan Wu, Kevin S. Chen +12
When Isaac Newton discovered the law of gravitation, he did so through an iterative process of analyzing observed data such as planetary patterns, finding the underlying mechanisms…
Efficient Test-Time Adaptation through Human-AI Interaction
Zora Zhiruo Wang, Apurva Gandhi, Rulin Shao +22
AI agents are trained on population-scale data to encode broad capabilities spanning those of many practitioners. Yet the artifacts they produce rarely meet the personal bar profes…
PACE: A Proxy for Agentic Capability Evaluation
Yueqi Song, Lintang Sutawika, Jiarui Liu +8
Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars…
Humans are Missing from AI Coding Agent Research
Zora Z. Wang, John Yang, Kilian Lieret +10
Recent progress in AI coding agent research has led to rapid improvements in agents' ability to autonomously perform complex software engineering tasks, from editing large codebase…
PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks
Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar +6
Creating and editing slides is a rich, multimodal activity that is ubiquitous in professional and educational settings, making it an ideal testbed for real-world computer-use agent…
Discretizing Reward Models
Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika +4
Despite their widespread use, the role of reward models in shaping reinforcement learning is poorly understood. Reward models offer a tempting promise: they automatically estimate…
Enhancing Software Engineering Through Closed-Loop Memory Optimization
Xuehang Guo, Zora Zhiruo Wang, Qingyun Wang +2
Large language models (LLMs) have enabled powerful software engineering (SE) agents capable of navigating complex codebases and resolving real-world issues. However, these agents r…
Recursive Agent Optimization
Apurva Gandhi, Satyaki Chakraborty, Xiangjun Wang +2
We introduce Recursive Agent Optimization (RAO), a reinforcement learning approach for training recursive agents: agents that can spawn and delegate sub-tasks to new instantiations…
Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks
Sanidhya Vijayvargiya, Vijay Viswanathan, Graham Neubig
Humans often specify tasks incompletely, so assistants must know when and how to ask clarifying questions. However, effective clarification remains challenging in software engineer…
Gym-Anything: Turn any Software into an Agent Environment
Pranjal Aggarwal, Graham Neubig, Sean Welleck
Computer-use agents hold the promise of assisting in a wide range of digital economic activities. However, current research has largely focused on short-horizon tasks over a limite…
CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents
Lintang Sutawika, Aditya Bharat Soni, Bharath Sriraam R R +8
A prerequisite for coding agents to perform tasks on large repositories is code localization - the identification of relevant files, classes, and functions to work on. While reposi…
CUBE: A Standard for Unifying Agent Benchmarks
Alexandre Lacoste, Nicolas Gontier, Oleh Shliazhko +23
The proliferation of agent benchmarks has created critical fragmentation that threatens research productivity. Each new benchmark requires substantial custom integration, creating…
How Well Does Agent Development Reflect Real-World Work?
Zora Zhiruo Wang, Sanidhya Vijayvargiya, Aspen Chen +7
AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor marke…
A Rubric-Supervised Critic from Sparse Real-World Outcomes
Xingyao Wang, Valerie Chen, Heng Ji +1
Academic benchmarks for coding agents tend to reward autonomous task completion, measured by verifiable rewards such as unit-test success. In contrast, real-world coding agents ope…
Effective Strategies for Asynchronous Software Engineering Agents
Jiayi Geng, Graham Neubig
AI agents have become increasingly capable at isolated software engineering (SWE) tasks such as resolving issues on Github. Yet long-horizon tasks involving multiple interdependent…
Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Xuhui Zhou, Weiwei Sun, Qianou Ma +8
As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user…
Hybrid-Gym: Training Coding Agents to Generalize Across Tasks
Yiqing Xie, Emmy Liu, Gaokai Zhang +7
When assessing the quality of coding agents, predominant benchmarks focus on solving single issues on GitHub, such as SWE-Bench. In contrast, in real use, these agents solve more v…
Modeling Distinct Human Interaction in Web Agents
Faria Huq, Zora Zhiruo Wang, Zhanqiu Guo +6
Despite rapid progress in autonomous web agents, human involvement remains essential for shaping preferences and correcting agent behavior as tasks unfold. However, current agentic…
Training Versatile Coding Agents in Synthetic Environments
Yiqi Zhu, Apurva Gandhi, Graham Neubig
Prior works on training software engineering agents have explored utilizing existing resources such as issues on GitHub repositories to construct software engineering tasks and cor…