6 papers
Distilling Reasoning Traces into Advisory Prompts for Software Engineering Tasks
Faizan Faisal, Prem Devanbu, Toufique Ahmed
Language models are widely used for generating and otherwise processing code (e.g., identifying code hallucinations, possible inputs, or predicting outputs); however, LLMs can make…
How Robustly do LLMs Understand Execution Semantics?
Claudio Spiess, Prem Devanbu, Earl T. Barr
LLMs demonstrate remarkable reasoning capabilities, yet whether they utilize internal world models or rely on sophisticated pattern matching remains open. We study LLMs through the…
On LLMs' Internal Representation of Code Correctness
Francisco Ribeiro, Claudio Spiess, Prem Devanbu +1
Despite the effectiveness of large language models (LLMs) for code generation, they often output incorrect code. One reason is that model output probabilities are often not well-co…
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
Ali Al-Kaswan, Claudio Spiess, Prem Devanbu +2
Large language models are increasingly used for code generation and debugging, but their outputs can still contain bugs, that originate from training data. Distinguishing whether a…
Localized Calibrated Uncertainty in Code Language Models
David Gros, Prem Devanbu
Large Language models (LLMs) can generate complicated source code from natural language prompts. However, LLMs can generate output that deviates from what the user wants, requiring…
Does In-IDE Calibration of Large Language Models work at Scale?
Roham Koohestani, Agnia Sergeyuk, David Gros +4
The introduction of large language models into integrated development environments (IDEs) is revolutionizing software engineering, yet it poses challenges to the usefulness and rel…