3 papers
cs.SE2025
Disproving Program Equivalence with LLMs
Miltiadis Allamanis, Pengcheng Yin
To evaluate large language models (LLMs) for code, research has used manually created unit test-based benchmarks. However, these tests are often inadequate, missing corner cases an…
cs.SE2024
Unsupervised Evaluation of Code LLMs with Round-Trip Correctness
Miltiadis Allamanis, Sheena Panthaplackel, Pengcheng Yin
To evaluate code large language models (LLMs), research has relied on a few small manually curated benchmarks, such as HumanEval and MBPP, which represent a narrow part of the real…
cs.LG2024
NExT: Teaching Large Language Models to Reason about Code Execution
Ansong Ni, Miltiadis Allamanis, Arman Cohan +4
A fundamental skill among human developers is the ability to understand and reason about program execution. As an example, a programmer can mentally simulate code execution in natu…