3 papers
cs.LG2026
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
Mohit Raghavendra, Soham Dan, Miguel Romero Calvo +12
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refac…
cs.AI2026
HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?
Tu Trinh, Mohamed Elfeki, Guangze Luo +9
Frontier coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous. The bottleneck is not raw capability, but judgm…
cs.SE2025
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Xiang Deng, Jeff Da, Edwin Pan +19
We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, compl…