5 papers
Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
Changshu Liu, Alireza Ghazanfari, Yang Chen +1
Code reasoning tasks are becoming prevalent in large language model (LLM) assessments. Yet, there is a dearth of studies on the impact of real-world complexities on code reasoning,…
Evaluating LLMs Code Reasoning Under Real-World Context
Changshu Liu
Code reasoning tasks are increasingly crucial to evaluating large language models (LLMs). Yet most existing benchmarks rely on simplistic, LLM-generated snippets or human-written s…
Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Models
Changshu Liu, Yang Chen, Reyhaneh Jabbarvand
This paper proposes CES, a task to evaluate the abilities of LLMs in simulating program execution and using that reasoning in programming tasks. Besides measuring the correctness o…
CodeMind: Evaluating Large Language Models for Code Reasoning
Changshu Liu, Yang Chen, Reyhaneh Jabbarvand
Large Language Models (LLMs) have been widely used to automate programming tasks. Their capabilities have been evaluated by assessing the quality of generated code through tests or…
A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models
Changshu Liu, Reyhaneh Jabbarvand
Code Executing Reasoning is becoming a new non-functional metric that assesses the ability of large language models (LLMs) in programming tasks. State-of-the-art frameworks (CodeMi…