9 papers · 1 filter
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
Shuangjie Yao, Hao Wang, Koushik Sen +3
Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all existing benchmarks still rely on t…
Trustworthy AI Software Engineers
Aldeida Aleti, Baishakhi Ray, Rashina Hoda +1
With the rapid rise of AI coding agents, the fundamental premise of what it means to be a software engineer is in question. In this vision paper, we examine what it means for an AI…
AppForge: From Assistant to Independent Developer -- Are GPTs Ready for Software Development?
Dezhi Ran, Yuan Cao, Mengzhou Wu +10
Large language models (LLMs) have demonstrated remarkable capability in function-level code generation tasks. Unlike isolated functions, real-world applications demand reasoning ov…
REFINE: Enhancing Program Repair Agents through Context-Aware Patch Refinement
Anvith Pabba, Simin Chen, Alex Mathai +2
Large Language Models (LLMs) have recently shown strong potential in automatic program repair (APR), especially in repository-level settings where the goal is to generate patches b…
Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities
Simin Chen, Yixin He, Suman Jana +1
LLM-based agents are increasingly deployed for software maintenance tasks such as automated program repair (APR). APR agents automatically fetch GitHub issues and use backend LLMs…
SemAgent: A Semantics Aware Program Repair Agent
Anvith Pabba, Alex Mathai, Anindya Chakraborty +1
Large Language Models (LLMs) have shown impressive capabilities in downstream software engineering tasks such as Automated Program Repair (APR). In particular, there has been a lot…
Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action
Saurabh Pujar, Ira Ceka, Irene Manotas +3
The rise of large language models (LLMs) has led to dramatic improvements across a wide range of natural language tasks. Their performance on certain tasks can be further enhanced…
CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning
Monoshi Kumar Roy, Simin Chen, Benjamin Steenhoek +4
Understanding and reasoning about code semantics is essential for enhancing code LLMs' abilities to solve real-world software engineering (SE) tasks. Although several code reasonin…
Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
Simin Chen, Pranav Pusarla, Baishakhi Ray
The rapid evolution of code largelanguage models underscores the need for effective and transparent benchmarking of their reasoning capabilities. However, the current benchmarking…