17 citations · 46 across the 12 of their papers we have counts for
13 papers
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation
Priyam Sahoo, Gaurav Mittal, Xiaomin Li +4
Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a…
Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
Spandan Garg, Benjamin Steenhoek, Yufan Huang
Current benchmarks for evaluating software engineering agents, such as SWE-Bench Verified, are predominantly derived from GitHub issues and fail to accurately reflect how developer…
CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning
Monoshi Kumar Roy, Simin Chen, Benjamin Steenhoek +4
Understanding and reasoning about code semantics is essential for enhancing code LLMs' abilities to solve real-world software engineering (SE) tasks. Although several code reasonin…
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
Benjamin Steenhoek, Michele Tufano, Neel Sundaresan +1
Software testing is a crucial but time-consuming aspect of software development, and recently, Large Language Models (LLMs) have gained popularity for automated test case generatio…
To Err is Machine: Vulnerability Detection Challenges LLM Reasoning
Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy +5
In this paper, we present a challenging code reasoning task: vulnerability detection. Large Language Models (LLMs) have shown promising results in natural-language and math reasoni…
Do Language Models Learn Semantics of Code? A Case Study in Vulnerability Detection
Benjamin Steenhoek, Md Mahbubur Rahman, Shaila Sharmin +1
Recently, pretrained language models have shown state-of-the-art performance on the vulnerability detection task. These models are pretrained on a large corpus of source code, then…
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
Benjamin Steenhoek, Michele Tufano, Neel Sundaresan +1
Software testing is a crucial aspect of software development, and the creation of high-quality tests that adhere to best practices is essential for effective maintenance. Recently,…
Reproducing Failures in Fault Signatures
Ashwin Kallingal Joshy, Benjamin Steenhoek, Xiuyuan Guo +1
Software often fails in the field, however reproducing and debugging field failures is very challenging: the failure-inducing input may be missing, and the program setup can be com…
TRACED: Execution-aware Pre-training for Source Code
Yangruibo Ding, Ben Steenhoek, Kexin Pei +3
Most existing pre-trained language models for source code focus on learning the static code text, typically augmented with static code structures (abstract syntax tree, dependency…
A Study of Static Warning Cascading Tools (Experience Paper)
Xiuyuan Guo, Ashwin Kallingal Joshy, Benjamin Steenhoek +2
Static analysis is widely used for software assurance. However, static analysis tools can report an overwhelming number of warnings, many of which are false positives. Applying sta…
An Empirical Study of Deep Learning Models for Vulnerability Detection
Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles +1
Deep learning (DL) models of code have recently reported great progress for vulnerability detection. In some cases, DL-based models have outperformed static analysis tools. Althoug…
Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability Detection
Benjamin Steenhoek, Hongyang Gao, Wei Le
Deep learning-based vulnerability detection has shown great performance and, in some studies, outperformed static analysis tools. However, the highest-performing approaches use tok…
Validating Static Warnings via Testing Code Fragments
Ashwin Kallingal Joshy, Xueyuan Chen, Benjamin Steenhoek +1
Static analysis is an important approach for finding bugs and vulnerabilities in software. However, inspecting and confirming static warnings are challenging and time-consuming. In…