17 papers
ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty
Yuyang Dai, Yuting Tang, Futoshi Futami +1
Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To…
Jumping the Line: Exploiting Length Predictions in LLM Scheduling
Yuyang Dai, Rana Shahout, Mahmood Sharif
Efficient request scheduling is increasingly important for reducing completion time in large language model (LLM) serving. Size-based policies such as Shortest Job First prioritize…
Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
Yuyang Dai, Bofei Huang, Hongbo Zhang +1
Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they have already identified correc…
Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?
Yuyang Dai, Xueqing Peng, Yuxia Wang +2
Large language models are increasingly applied as autonomous decision-making agents. However, in executive business decisions, existing benchmarks are limited to textonly settings.…
Overview of FinMMEval 2026 Task 2: Multilingual Financial Short-Answer Question Answering
Zhuohan Xie, Xueqing Peng, Georgi Georgiev +18
FinMMEval 2026 Task 2 evaluates short-answer financial question answering over multilingual evidence. Each final-test item pairs an English question with financial statements and n…
Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering
Zhuohan Xie, Yuyang Dai, Rania Elbadry +18
FinMMEval 2026 Task 1 evaluates multilingual financial multiple-choice question answering in English, Chinese, Arabic, and Hindi. The task tests whether systems can select the corr…
AI YOU Town: Make Friends and Money with Your Digital Twin
Yan Lin, Yuyang Dai, Jiahui Geng +1
Existing approaches to infer user traits and generate responses consistent with a persona rely on static prompting. They lack calibrated uncertainty, ignore sequential evidence, an…
Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation
Yuyang Dai, Xueqing Peng, Lingfei Qian +1
Evaluating the decision-making capabilities of large language models (LLMs) is a growing research priority, yet existing benchmarks focus on isolated cognitive tasks such as reason…
Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypot
Yuyang Dai, Yushun Dong
Large language models deployed as commercial APIs are vulnerable to model extraction attacks, while existing defenses either act too late or degrade utility for legitimate users. W…
Herculean: An Agentic Benchmark for Financial Intelligence
Xueqing Peng, Zhuohan Xie, Yupeng Cao +60
As AI agents advance, the central question is no longer whether they can solve isolated financial tasks, but whether their capabilities generalize across complete professional work…
GraphIP-Bench: How Hard Is It to Steal a Graph Neural Network, and Can We Stop It?
Kaixiang Zhao, Bolin Shen, Yuyang Dai +2
Graph neural networks (GNNs) deployed as cloud services can be stolen through model-extraction attacks, which train a surrogate from query responses to reproduce the target's behav…
Does Your Wildfire Prediction Model Actually Work, or Just Score Well?
Yangshuang Xu, Yuyang Dai, Liling Chang +2
Wildfire prediction is important for early warning and resource allocation, yet existing Earth foundation models (Earth FMs) are pretrained for general atmospheric and geophysical…
FinReporting: An Agentic Workflow for Localized Reporting of Cross-Jurisdiction Financial Disclosures
Fan Zhang, Mingzi Song, Rania Elbadry +20
Financial reporting systems increasingly leverage Large Language Models (LLMs) to extract and summarize corporate disclosures. However, most existing approaches assume a single-mar…
EpiGraph: Building Generalists for Evidence-Intensive Epilepsy Reasoning in the Wild
Yuyang Dai, Zheng Chen, Jathurshan Pradeepkumar +4
Epilepsy diagnosis and treatment require evidence-intensive reasoning across heterogeneous clinical knowledge, including biosignal patterns, genetic mechanisms, pharmacogenomics, t…
Beyond 0-100: How Confidence Scale Design Shapes Verbalized Confidence and LLM Metacognition
Yuyang Dai, Yuxia Wang
Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0-…
The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems
Zhuohan Xie, Rania Elbadry, Fan Zhang +12
We present the setup and the tasks of the FinMMEval Lab at CLEF 2026, which introduces the first multilingual and multimodal evaluation framework for financial Large Language Model…
RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?
Yuyang Dai, Yan Lin, Zhuohan Xie +1
Reliable financial reasoning requires knowing not only how to answer, but also when an answer cannot be justified. In real financial practice, problems often rely on implicit assum…