4 papers
How Robustly do LLMs Understand Execution Semantics?
Claudio Spiess, Prem Devanbu, Earl T. Barr
LLMs demonstrate remarkable reasoning capabilities, yet whether they utilize internal world models or rely on sophisticated pattern matching remains open. We study LLMs through the…
On LLMs' Internal Representation of Code Correctness
Francisco Ribeiro, Claudio Spiess, Prem Devanbu +1
Despite the effectiveness of large language models (LLMs) for code generation, they often output incorrect code. One reason is that model output probabilities are often not well-co…
Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs
Ali Al-Kaswan, Claudio Spiess, Prem Devanbu +2
Large language models are increasingly used for code generation and debugging, but their outputs can still contain bugs, that originate from training data. Distinguishing whether a…
Does In-IDE Calibration of Large Language Models work at Scale?
Roham Koohestani, Agnia Sergeyuk, David Gros +4
The introduction of large language models into integrated development environments (IDEs) is revolutionizing software engineering, yet it poses challenges to the usefulness and rel…