9 papers
PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
Shuo Lin, Jinzhe Li, Chenlu Guo +3
Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answ…
More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models
Tianxiang Gao, Jinzhe Li, Zhiyuan Li +2
Direct-decision models enable efficient classification and evaluation by selecting from predefined candidates, but their reliability depends on more than prediction accuracy. Acros…
MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs
Jinzhe Li, Gengxu Li, Jinnan Li +2
As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructions but also on validating th…
MerLean-Prover: A Recursive Looping Harness for Lean 4 Theorem Proving
Jinzheng Li, Zeru Zhu, Yuanjie Ren
MerLean-Prover is an end-to-end Lean4 theorem prover that replaces sorry declarations with kernel-checkable proofs. It is built from three agent types (Planning, Check, and Lean) c…
Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability
Haiqi Yang, Jinzhe Li, Gengxu Li +2
Large Multimodal Models (LMMs) have witnessed remarkable growth, showcasing formidable capabilities in handling intricate multimodal tasks with exceptional performance. Recent rese…
Refining Critical Thinking in LLM Code Generation: A Faulty Premise-based Evaluation Framework
Jialin Li, Jinzhe Li, Gengxu Li +2
With the advancement of code generation capabilities in large language models (LLMs), their reliance on input premises has intensified. When users provide inputs containing faulty…
Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models
Jinzhe Li, Gengxu Li, Yi Chang +1
Large language models (LLMs) have witnessed rapid advancements, demonstrating remarkable capabilities. However, a notable vulnerability persists: LLMs often uncritically accept fla…
JurisCTC: Enhancing Legal Judgment Prediction via Cross-Domain Transfer and Contrastive Learning
Zhaolu Kang, Hongtian Cai, Xiangyang Ji +2
In recent years, Unsupervised Domain Adaptation (UDA) has gained significant attention in the field of Natural Language Processing (NLP) owing to its ability to enhance model gener…
StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following
Jinnan Li, Jinzhe Li, Yue Wang +2
Multi-turn instruction following capability constitutes a core competency of large language models (LLMs) in real-world applications. Existing evaluation benchmarks predominantly f…