6 papers
The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning
Yubo Li, Lu Zhang, Tianchong Jiang +2
Large language models fail when a salient surface cue conflicts with an unstated feasibility constraint. We introduce the Heuristic Override Benchmark (HOB): 500 instances spanning…
Toward Functional and Non-Functional Evaluation of Application-Level Code Generation
Ruwei Pan, Yakun Zhang, Qingyuan Liang +4
Large language models (LLMs) have achieved strong performance on code generation. However, most prior evaluations focus on snippet-level outputs, such as function generation or rep…
TRACE: A Framework for Analyzing and Enhancing Stepwise Reasoning in Vision-Language Models
Shima Imani, Seungwhan Moon, Lambert Mathias +2
Reliable mathematical and scientific reasoning remains an open challenge for large vision-language models. Standard final-answer evaluation often masks reasoning errors, allowing s…
SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Code
Shima Imani, Seungwhan Moon, Adel Ahmadyan +3
We introduce, a large-scale synthetic benchmark of 15,045 university-level physics problems (90/10% train/test split). Each problem is fully parameterized, supporting an effectivel…
PRiSM: An Agentic Multimodal Benchmark for Scientific Reasoning via Python-Grounded Evaluation
Shima Imani, Seungwhan Moon, Adel Ahmadyan +3
Evaluating vision-language models (VLMs) in scientific domains like mathematics and physics poses unique challenges that go far beyond predicting final answers. These domains deman…
Step-Wise Formal Verification for LLM-Based Mathematical Problem Solving
Kuo Zhou, Lu Zhang
Large Language Models (LLMs) have demonstrated formidable capabilities in solving mathematical problems, yet they may still commit logical reasoning and computational errors during…