Publications (16)
How Robustly do LLMs Understand Execution Semantics?
Claudio Spiess, Prem Devanbu, Earl T. Barr
LLMs demonstrate remarkable reasoning capabilities, yet whether they utilize internal world models or rely on sophisticated pattern matching remains open. We study LLMs through the…
Localized Calibrated Uncertainty in Code Language Models
David Gros, Prem Devanbu
Large Language models (LLMs) can generate complicated source code from natural language prompts. However, LLMs can generate output that deviates from what the user wants, requiring…
A Survey of Trojans in Neural Models of Source Code: Taxonomy and Techniques
Aftab Hussain, Md Rafiqul Islam Rabin, Toufique Ahmed +4
In this work, we study literature in Explainable AI and Safe AI to understand poisoning of neural models of code. In order to do so, we first establish a novel taxonomy for Trojan…
Towards Understanding What Code Language Models Learned
Toufique Ahmed, Dian Yu, Chengxuan Huang +3
Pre-trained language models are effective in a variety of natural language tasks, but it has been argued their capabilities fall short of fully learning meaning or understanding la…
Studying the Difference Between Natural and Programming Language Corpora
Casey Casalnuovo, Kenji Sagae, Prem Devanbu
Code corpora, as observed in large software systems, are now known to be far more repetitive and predictable than natural language corpora. But why? Does the difference simply aris…
Deep Learning & Software Engineering: State of Research and Future Directions
Prem Devanbu, Matthew Dwyer, Sebastian Elbaum +6
Given the current transformative potential of research that sits at the intersection of Deep Learning (DL) and Software Engineering (SE), an NSF-sponsored community workshop was co…
Does In-IDE Calibration of Large Language Models work at Scale?
Roham Koohestani, Agnia Sergeyuk, David Gros +4
The introduction of large language models into integrated development environments (IDEs) is revolutionizing software engineering, yet it poses challenges to the usefulness and rel…
AI Safety Subproblems for Software Engineering Researchers
David Gros, Prem Devanbu, Zhou Yu
In this 4-page manuscript we discuss the problem of long-term AI Safety from a Software Engineering (SE) research viewpoint. We briefly summarize long-term AI Safety, and the chall…
Rebuttal to Berger et al., TOPLAS 2019
Baishakhi Ray, Prem Devanbu, Vladimir Filkov
Berger et al., published in TOPLAS 2019, is a critique of our 2014 FSE conference abstract and its archival version, the 2017 CACM paper: A Large-Scale Study of Programming Languag…
Do People Prefer "Natural" code?
Casey Casalnuovo, Kevin Lee, Hulin Wang +2
Natural code is known to be very repetitive (much more so than natural language corpora); furthermore, this repetitiveness persists, even after accounting for the simpler syntax of…
Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time
Razvan Mihai Popescu, David Gros, Andrei Botocan +3
The rise of large language models for code has reshaped software development. Autonomous coding agents, able to create branches, open pull requests, and perform code reviews, now a…
On LLMs' Internal Representation of Code Correctness
Francisco Ribeiro, Claudio Spiess, Prem Devanbu +1
Despite the effectiveness of large language models (LLMs) for code generation, they often output incorrect code. One reason is that model output probabilities are often not well-co…
Code to Comment "Translation": Data, Metrics, Baselining & Evaluation
David Gros, Hariharan Sezhiyan, Prem Devanbu +1
The relationship of comments to code, and in particular, the task of generating useful comments given the code, has long been of interest. The earliest approaches have been based o…
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
Ali Al-Kaswan, Claudio Spiess, Prem Devanbu +2
Large language models are increasingly used for code generation and debugging, but their outputs can still contain bugs, that originate from training data. Distinguishing whether a…
Distilling Reasoning Traces into Advisory Prompts for Software Engineering Tasks
Faizan Faisal, Prem Devanbu, Toufique Ahmed
Language models are widely used for generating and otherwise processing code (e.g., identifying code hallucinations, possible inputs, or predicting outputs); however, LLMs can make…
Calibration and Correctness of Language Models for Code
Claudio Spiess, David Gros, Kunal Suresh Pai +6
Machine learning models are widely used, but can also often be wrong. Users would benefit from a reliable indication of whether a given output from a given model should be trusted,…