14 papers
PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
Shuo Lin, Jinzhe Li, Chenlu Guo +3
Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answ…
More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models
Tianxiang Gao, Jinzhe Li, Zhiyuan Li +2
Direct-decision models enable efficient classification and evaluation by selecting from predefined candidates, but their reliability depends on more than prediction accuracy. Acros…
ConflictVLA-Bench: Benchmarking Behavioral Responses of Vision-Language-Action Models to Premise Conflicts
Liyu Hou, Yuan Wu, Yi Chang
While Vision-Language-Action (VLA) models perform strongly on manipulation tasks, their responses to invalid task premises remain underexplored. Existing evaluations of premise con…
What if LLMs Ate Their Words: Causal History Effects in Multi-Turn Interaction
Jinnan Li, Zheren Fu, Yue Wang +3
Multi-turn interaction creates a feedback process in which an LLM's previous responses become context for later behavior. Prior work shows substantial multi-turn degradation and th…
OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding
Gengxu Li, Yuan Wu, Yi Chang
Text-rich image understanding requires multimodal large language models (MLLMs) to organize OCR (Optical Character Recognition)-grounded evidence across words, layout, fields, char…
MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs
Jinzhe Li, Gengxu Li, Jinnan Li +2
As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructions but also on validating th…
Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
Yuan Wu, Zongxian Yang, Jiayu Qian +5
Large vision-language models (VLMs) often benefit from chain-of-thought (CoT) prompting in general domains, yet its efficacy in medical vision-language tasks remains underexplored.…
ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructions
Jialin Li, Yuan Wu, Yi Chang
To integrate seamlessly into real-world software engineering, Code Agents must evolve from passive instruction followers into proactive collaborative partners. However, current eva…
ConfProBench: A Confidence Evaluation Benchmark for MLLM-Based Process Judges
Yue Zhou, Yi Chang, Yuan Wu
Reasoning is a critical capability of multimodal large language models (MLLMs) for solving complex multimodal tasks, and judging the correctness of reasoning steps is crucial for i…
Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability
Haiqi Yang, Jinzhe Li, Gengxu Li +2
Large Multimodal Models (LMMs) have witnessed remarkable growth, showcasing formidable capabilities in handling intricate multimodal tasks with exceptional performance. Recent rese…
Refining Critical Thinking in LLM Code Generation: A Faulty Premise-based Evaluation Framework
Jialin Li, Jinzhe Li, Gengxu Li +2
With the advancement of code generation capabilities in large language models (LLMs), their reliance on input premises has intensified. When users provide inputs containing faulty…
Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models
Jinzhe Li, Gengxu Li, Yi Chang +1
Large language models (LLMs) have witnessed rapid advancements, demonstrating remarkable capabilities. However, a notable vulnerability persists: LLMs often uncritically accept fla…
Asymmetric Co-Training for Source-Free Few-Shot Domain Adaptation
Gengxu Li, Yuan Wu
Source-free unsupervised domain adaptation (SFUDA) has gained significant attention as an alternative to traditional unsupervised domain adaptation (UDA), which relies on the const…
Length-Controlled Margin-Based Preference Optimization without Reference Model
Gengxu Li, Tingyu Xia, Yi Chang +1
Direct Preference Optimization (DPO) is a widely adopted offline algorithm for preference-based reinforcement learning from human feedback (RLHF), designed to improve training simp…