7 papers
Terminus-4B: Can a Smaller Model Replace Frontier LLMs at Agentic Execution Tasks?
Spandan Garg, Vikram Nitin, Yufan Huang
Modern coding agents increasingly delegate specialized subtasks to subagents, which are smaller, focused agentic loops that handle narrow responsibilities like search, debugging or…
Debug2Fix: Can Interactive Debugging Help Coding Agents Fix More Bugs?
Spandan Garg, Yufan Huang
While significant progress has been made in automating various aspects of software development through coding agents, there is still significant room for improvement in their bug f…
Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
Spandan Garg, Benjamin Steenhoek, Yufan Huang
Current benchmarks for evaluating software engineering agents, such as SWE-Bench Verified, are predominantly derived from GitHub issues and fail to accurately reflect how developer…
PerfBench: Can Agents Resolve Real-World Performance Bugs?
Spandan Garg, Roshanak Zilouchian Moghaddam, Neel Sundaresan
Performance bugs are inefficiencies in software that waste computational resources without causing functional failures, making them particularly challenging to detect and fix. Whil…
The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
Shanchao Liang, Spandan Garg, Roshanak Zilouchian Moghaddam
As large language models (LLMs) become increasingly capable and widely adopted, benchmarks play a central role in assessing their practical utility. For example, SWE-Bench Verified…
RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code
Dhruv Gautam, Spandan Garg, Jinu Jang +2
Recent advances in language model (LM) agents and function calling have enabled autonomous, feedback-driven systems to solve problems across various digital domains. To better unde…