12 papers
PowerBench: A Benchmark for Agentic Retrieval and Reasoning in Power Systems
Xijing Wang, Yinsheng Yao, Jinru Ding +5
Large language model (LLM) agents offer new opportunities for automated analysis in industry. However, rigorous evaluation of such agents-for example, within power system scenarios…
P2E-VQ: ECG-linked representation augmentation for PPG via discrete patch retrieval
Zhongli Wu, Zhuangzhi Gao, He Zhao +8
Photoplethysmography (PPG) is widely used in consumer wearables because of its low cost and ease of acquisition. However, unlike electrocardiography (ECG), PPG measures peripheral…
MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
Runhan Shi, Quan Zhou, Yuqian Xu +14
Large language models (LLMs) are increasingly deployed in online medical consultation, yet existing benchmarks remain poorly aligned with real clinical practice. Many rely on synth…
MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Models
Jinru Ding, Chuchu Jiang, Lu Lu +12
Existing medical AI benchmarks lack process visibility, atomic skill evaluation, and integrated hallucination detection. We introduce MedBench v5, a redesigned benchmark for clinic…
SafeMed-R1: Clinician-Audited Safety and Ethics Alignment for Medical Large Language Models
Chao Ding, Mouxiao Bian, Tianbin Li +12
Large language models(LLMs) increasingly match expert performance on licensing examinations, yet routine clinical use remains limited because governance requires auditable reasonin…
FinDocMRE: A Benchmark for Document-Level Financial Multimodal Reasoning Evaluation
Jiayong Zhu, Jiangtong Li, Jinru Ding +3
While Large Multimodal Models (LMMs) excel in general visual tasks, their deployment in specialized financial contexts remains insufficient. Existing benchmarks prioritize isolated…
FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting
Yiyun Zhu, Yidong Jiang, Ziwen Xu +4
Large language models (LLMs) are increasingly deployed in financial research workflows, where their role is evolving from single-model assistance for human analysts toward autonomo…
EpiPlanAgent: Agentic Automated Epidemic Response Planning
Kangkun Mao, Fang Xu, Jinru Ding +8
Epidemic response planning is essential yet traditionally reliant on labor-intensive manual methods. This study aimed to design and evaluate EpiPlanAgent, an agent-based system usi…
Beyond Knowledge to Agency: Evaluating Expertise, Autonomy, and Integrity in Finance with CNFinBench
Jinru Ding, Chao Ding, Yidong Jiang +9
As large language models (LLMs) become high-privilege agents in risk-sensitive settings, they introduce systemic threats beyond hallucination, where minor compliance errors can cau…
MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
Jinru Ding, Lu Lu, Chao Ding +15
Recent advances in medical large language models (LLMs), multimodal models, and agents demand evaluation frameworks that reflect real clinical workflows and safety constraints. We…
MedCalc-Eval and MedCalc-Env: Advancing Medical Calculation Capabilities of Large Language Models
Kangkun Mao, Jinru Ding, Jiayuan Chen +6
As large language models (LLMs) enter the medical domain, most benchmarks evaluate them on question answering or descriptive reasoning, overlooking quantitative reasoning critical…
RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Model
Zhuangzhi Gao, Hongyi Qin, He Zhao +10
Retinal biomarkers captured by color fundus photography and optical coherence tomography provide clinically valuable evidence for both ocular and systemic diseases. Multimodal larg…