3 papers
cs.SE2025
SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks
Sanket Mhatre, Yasharth Bajpai, Sumit Gulwani +2
AI coding agents have shown great progress on Python software engineering benchmarks like SWE-Bench, and for other languages like Java and C in benchmarks like Multi-SWE-Bench. How…
cs.CL2025
ConDABench: Interactive Evaluation of Language Models for Data Analysis
Avik Dutta, Priyanshu Gupta, Hosein Hasanbeig +6
Real-world data analysis tasks often come with under-specified goals and unclean data. User interaction is necessary to understand and disambiguate a user's intent, and hence, esse…
cs.SE2025
Why AI Agents Still Need You: Findings from Developer-Agent Collaborations in the Wild
Aayush Kumar, Yasharth Bajpai, Sumit Gulwani +2
Software Engineering Agents (SWE agents) can autonomously perform development tasks on benchmarks like SWE Bench, but still face challenges when tackling complex and ambiguous real…