14 papers
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
Shuangjie Yao, Hao Wang, Koushik Sen +3
Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all existing benchmarks still rely on t…
Cost-Efficient Theorem Proving via Agent Orchestration in Program Verification
Shuangjie Yao, Nikolaus Holzer, Mark Paul Santolucito +3
Program verification establishes software correctness through machine-checkable proofs constructed in theorem provers. It's a guarantee especially valuable for code generated by la…
Adversarial Testing of Automated Program Repair Agents for Security Vulnerabilities
Fares Trad, Simin Chen, Hung Viet Pham +2
Software agents with Large Language Models (LLMs) are designed for Automated Program Repair (APR) tasks, raising the possibility that, in the near future, APR agents will fix bugs…
Trustworthy AI Software Engineers
Aldeida Aleti, Baishakhi Ray, Rashina Hoda +1
With the rapid rise of AI coding agents, the fundamental premise of what it means to be a software engineer is in question. In this vision paper, we examine what it means for an AI…
Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers
Simin Chen, Jinjun Peng, Yixin He +2
Deep learning (DL) compilers are core infrastructure in modern DL systems, offering flexibility and scalability beyond vendor-specific libraries. This work uncovers a fundamental v…
AppForge: From Assistant to Independent Developer -- Are GPTs Ready for Software Development?
Dezhi Ran, Yuan Cao, Mengzhou Wu +10
Large language models (LLMs) have demonstrated remarkable capability in function-level code generation tasks. Unlike isolated functions, real-world applications demand reasoning ov…
REFINE: Enhancing Program Repair Agents through Context-Aware Patch Refinement
Anvith Pabba, Simin Chen, Alex Mathai +2
Large Language Models (LLMs) have recently shown strong potential in automatic program repair (APR), especially in repository-level settings where the goal is to generate patches b…
TempoBench: Reasoning Execution Without Causal Attribution Is Just Simulation
Nikolaus Holzer, William Fishell, Baishakhi Ray +1
Current training paradigms, optimized for long-horizon reasoning trace execution, have made Large Language Models (LLMs) excel at pattern matching and forward simulation of reasoni…
Red Teaming Program Repair Agents: When Correct Patches can Hide Vulnerabilities
Simin Chen, Yixin He, Suman Jana +1
LLM-based agents are increasingly deployed for software maintenance tasks such as automated program repair (APR). APR agents automatically fetch GitHub issues and use backend LLMs…
SemAgent: A Semantics Aware Program Repair Agent
Anvith Pabba, Alex Mathai, Anindya Chakraborty +1
Large Language Models (LLMs) have shown impressive capabilities in downstream software engineering tasks such as Automated Program Repair (APR). In particular, there has been a lot…
Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action
Saurabh Pujar, Ira Ceka, Irene Manotas +3
The rise of large language models (LLMs) has led to dramatic improvements across a wide range of natural language tasks. Their performance on certain tasks can be further enhanced…
CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning
Monoshi Kumar Roy, Simin Chen, Benjamin Steenhoek +4
Understanding and reasoning about code semantics is essential for enhancing code LLMs' abilities to solve real-world software engineering (SE) tasks. Although several code reasonin…
Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
Simin Chen, Pranav Pusarla, Baishakhi Ray
The rapid evolution of code largelanguage models underscores the need for effective and transparent benchmarking of their reasoning capabilities. However, the current benchmarking…
Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation
Simin Chen, Yiming Chen, Zexin Li +8
Data contamination has received increasing attention in the era of large language models (LLMs) due to their reliance on vast Internet-derived training corpora. To mitigate the ris…