7 papers
PACE: A Proxy for Agentic Capability Evaluation
Yueqi Song, Lintang Sutawika, Jiarui Liu +8
Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars…
How can we assess human-agent interactions? Case studies in software agent design
Valerie Chen, Rohit Malhotra, Xingyao Wang +7
While benchmarks measure the accuracy of LLM-powered agents, they mostly assume full automation, failing to represent the collaborative nature of real-world use cases. In this pape…
CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents
Lintang Sutawika, Aditya Bharat Soni, Bharath Sriraam R R +8
A prerequisite for coding agents to perform tasks on large repositories is code localization - the identification of relevant files, classes, and functions to work on. While reposi…
OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou +4
Recent advances in AI agents capable of solving complex, everyday tasks, from scheduling to customer service, have enabled deployment in real-world settings, but their possibilitie…
SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories
Aditya Bharat Soni, Rajat Ghosh, Vaishnavi Bhargava +2
Software testing is crucial for ensuring the correctness and reliability of software systems. Automated generation of issue reproduction tests from natural language issue descripti…
Coding Agents with Multimodal Browsing are Generalist Problem Solvers
Aditya Bharat Soni, Boxuan Li, Xingyao Wang +2
Modern human labor is characterized by specialization; we train for years and develop particular tools that allow us to perform well across a variety of tasks. In addition, AI agen…