4 papers
Does SWE-Bench-Verified Test Agent Ability or Model Memory?
Thanosan Prathifkumar, Noble Saji Mathews, Meiyappan Nagappan
SWE-Bench-Verified, a dataset comprising 500 issues, serves as a de facto benchmark for evaluating various large language models (LLMs) on their ability to resolve GitHub issues. B…
AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests
Lara Khatib, Noble Saji Mathews, Meiyappan Nagappan
Bug reproduction is critical in the software debugging and repair process, yet the majority of bugs in open-source and industrial settings lack executable tests to reproduce them a…
Is Your Automated Software Engineer Trustworthy?
Noble Saji Mathews, Meiyappan Nagappan
Large Language Models (LLMs) are being increasingly used in software engineering tasks, with an increased focus on bug report resolution over the past year. However, most proposed…
Design choices made by LLM-based test generators prevent them from finding bugs
Noble Saji Mathews, Meiyappan Nagappan
There is an increasing amount of research and commercial tools for automated test case generation using Large Language Models (LLMs). This paper critically examines whether recent…