4 papers
Evaluating LLMs Code Reasoning Under Real-World Context
Changshu Liu
Code reasoning tasks are increasingly crucial to evaluating large language models (LLMs). Yet most existing benchmarks rely on simplistic, LLM-generated snippets or human-written s…
Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
Changshu Liu, Alireza Ghazanfari, Yang Chen +1
Code reasoning tasks are becoming prevalent in large language model (LLM) assessments. Yet, there is a dearth of studies on the impact of real-world complexities on code reasoning,…
Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Models
Changshu Liu, Yang Chen, Reyhaneh Jabbarvand
This paper proposes CES, a task to evaluate the abilities of LLMs in simulating program execution and using that reasoning in programming tasks. Besides measuring the correctness o…
A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models
Changshu Liu, Reyhaneh Jabbarvand
Code Executing Reasoning is becoming a new non-functional metric that assesses the ability of large language models (LLMs) in programming tasks. State-of-the-art frameworks (CodeMi…