8 papers
The Bottleneck Is Retrieval, Not Reading: Diagnosing Evidence-Based Generation with RAG for Long Context Question Answering
Debopriyo Banerjee, Owais Aijaz, Ahmer Tabassum +3
Retrieval-augmented generation (RAG) is the dominant paradigm for grounding large language models in external corpora, but its similarity-driven, chunked pipeline has structural bl…
Overview of FinMMEval 2026 Task 2: Multilingual Financial Short-Answer Question Answering
Zhuohan Xie, Xueqing Peng, Georgi Georgiev +18
FinMMEval 2026 Task 2 evaluates short-answer financial question answering over multilingual evidence. Each final-test item pairs an English question with financial statements and n…
Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering
Zhuohan Xie, Yuyang Dai, Rania Elbadry +18
FinMMEval 2026 Task 1 evaluates multilingual financial multiple-choice question answering in English, Chinese, Arabic, and Hindi. The task tests whether systems can select the corr…
SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning
Rania Elbadry, Sarfraz Ahmad, Ahmed Heakl +12
English financial NLP has advanced rapidly through benchmarks targeting earnings analysis, market sentiment, tabular reasoning, and financial question answering, yet Arabic financi…
The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems
Zhuohan Xie, Rania Elbadry, Fan Zhang +12
We present the setup and the tasks of the FinMMEval Lab at CLEF 2026, which introduces the first multilingual and multimodal evaluation framework for financial Large Language Model…
EngTrace: A Symbolic Benchmark for Verifiable Process Evaluation of Engineering Reasoning
Ayesha Gull, Muhammad Usman Safder, Rania Elbadry +4
Most engineering problems involve complex multi-step derivations through checkable intermediate quantities. Grading only the final answer misjudges LLMs, crediting correct answers…
FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning
Zhuohan Xie, Daniil Orel, Rushil Thareja +22
Multi-step symbolic reasoning is essential for robust financial analysis; yet, current benchmarks largely overlook this capability. Existing datasets such as FinQA and ConvFinQA em…
Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs
Lang Gao, Kaiyang Wan, Wei Liu +6
Bias in Large Language Models (LLMs) significantly undermines their reliability and fairness. We focus on a common form of bias: when two reference concepts in the model's concept…