6 citations · 11 across the 6 of their papers we have counts for
3 papers · 1 filter
NExT: Teaching Large Language Models to Reason about Code Execution
Ansong Ni, Miltiadis Allamanis, Arman Cohan +4
A fundamental skill among human developers is the ability to understand and reason about program execution. As an example, a programmer can mentally simulate code execution in natu…
Grounding Data Science Code Generation with Input-Output Specifications
Yeming Wen, Pengcheng Yin, Kensen Shi +3
Large language models (LLMs) have recently demonstrated a remarkable ability to generate code from natural language (NL) prompts. However, in the real world, NL is often too ambigu…
Unsupervised Evaluation of Code LLMs with Round-Trip Correctness
Miltiadis Allamanis, Sheena Panthaplackel, Pengcheng Yin
To evaluate code large language models (LLMs), research has relied on a few small manually curated benchmarks, such as HumanEval and MBPP, which represent a narrow part of the real…