5 papers
Are LLMs Stable Formal Logic Translators in Logical Reasoning Across Linguistically Diversified Texts?
Qingchuan Li, Jiatong Li, Zirui Liu +4
Logical reasoning with large language models (LLMs) has received growing attention. One mainstream approach translates natural language into formal logic and then applies symbolic…
Generative Cognitive Diagnosis
Jiatong Li, Qi Liu, Mengxiao Zhu
Cognitive diagnosis (CD) models latent cognitive states of human learners by analyzing their response patterns on diagnostic tests, serving as a crucial machine learning technique…
am-ELO: A Stable Framework for Arena-based LLM Evaluation
Zirui Liu, Jiatong Li, Yan Zhuang +5
Arena-based evaluation is a fundamental yet significant evaluation paradigm for modern AI models, especially large language models (LLMs). Existing framework based on ELO rating sy…
Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach
Qingchuan Li, Jiatong Li, Tongxuan Liu +4
Large Language Models (LLMs) have exhibited remarkable potential across a wide array of reasoning tasks, including logical reasoning. Although massive efforts have been made to emp…
PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbations
Jiatong Li, Renjun Hu, Kunzhe Huang +5
Expert-designed close-ended benchmarks are indispensable in assessing the knowledge capacity of large language models (LLMs). Despite their widespread use, concerns have mounted re…