5 papers
Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation
Michele Tufano, James McClure, José Cambronero +7
LLM-based agents are increasingly used for coding tasks, where they have outperformed many classical approaches and scaled to repository-level tasks, such as test generation. Howev…
Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair
Runxiang Cheng, Michele Tufano, José Cambronero +5
Bug Reproduction Tests (BRTs) have been used in many Automated Program Repair (APR) systems, primarily for validating promising fixes and aiding fix generation. In practice, when d…
Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair
José Cambronero, Michele Tufano, Sherry Shi +7
Agentic Automated Program Repair (APR) is increasingly tackling complex, repository-level bugs in industry, but ultimately these patches still need to be reviewed by a human before…
From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineering
Tao Dong, Harini Sampath, Ja Young Lee +2
As Large Language Models (LLMs) evolve from code generators into collaborative partners for software engineers, our methods for evaluation are lagging. Current benchmarks, focused…
Towards a Human-in-the-Loop Framework for Reliable Patch Evaluation Using an LLM-as-a-Judge
Sherry Shi, Renyao Wei, Michele Tufano +4
Reliable evaluation is crucial for advancing Automated Program Repair (APR), but prevailing benchmarks rely on execution-based evaluation methods (unit test pass@k), which fail to…