7 citations · 39 across the 53 of their papers we have counts for
36 papers · 1 filter
Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models
Grzegorz Kulik, Mikołaj Pokrywka, Adam Jatowt +1
Dialectal machine translation remains challenging due to limited data and strong linguistic variation not captured by standard benchmarks, which often assume standardized and well-…
Large Language Models Systematically Favor Popular Options: Evidence and Mitigation Across MCQs
Abdelrahman Abdallah, Mohammed Ali, Bhawna Piryani +2
Multiple-choice questions (MCQs) are a standard format for evaluating large language models (LLMs), yet the popularity of answer options can confound evaluation. Modern LLMs system…
Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation
Lorenz Brehme, Adam Jatowt
Recent advances in LLMs and the adoption of RAG systems in industry have created a need for domain-specific question-answer datasets that can assess RAG performance on proprietary…
Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring
Jamshid Mozafari, Bhawna Piryani, Adam Jatowt
Estimating question difficulty is a critical component in evaluating and improving large language models (LLMs) for question answering (QA). Existing approaches often rely on reada…
Pretraining Exposure Explains Popularity Judgments in Large Language Models
Jamshid Mozafari, Bhawna Piryani, Adam Jatowt
Large language models (LLMs) exhibit systematic preferences for well-known entities, a phenomenon often attributed to popularity bias. However, the extent to which these preference…
Context Convergence Improves Answering Inferential Questions
Jamshid Mozafari, Bhawna Piryani, Adam Jatowt
While Large Language Models (LLMs) are widely used in open-domain Question Answering (QA), their ability to handle inferential questions-where answers must be derived rather than d…
How often do Answers Change? Estimating Recency Requirements in Question Answering
Bhawna Piryani, Zehra Mert, Adam Jatowt
Large language models (LLMs) often rely on outdated knowledge when answering time-sensitive questions, leading to confident yet incorrect responses. Without explicit signals indica…
PARSE: An Open-Domain Reasoning Question Answering Benchmark for Persian
Jamshid Mozafari, Seyed Parsa Mousavinasab, Adam Jatowt
Reasoning-focused Question Answering (QA) has advanced rapidly with Large Language Models (LLMs), yet high-quality benchmarks for low-resource languages remain scarce. Persian, spo…
Inferential Question Answering
Jamshid Mozafari, Hamed Zamani, Guido Zuccon +1
Despite extensive research on a wide range of question answering (QA) systems, most existing work focuses on answer containment-i.e., assuming that answers can be directly extracte…
Exploring NLP Benchmarks in an Extremely Low-Resource Setting
Ulin Nuha, Adam Jatowt
The effectiveness of Large Language Models (LLMs) diminishes for extremely low-resource languages, such as indigenous languages, primarily due to the lack of labeled data. Despite…
DeAR: Dual-Stage Document Reranking with Reasoning Agents via LLM Distillation
Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani +1
Large Language Models (LLMs) have transformed listwise document reranking by enabling global reasoning over candidate sets, yet single models often struggle to balance fine-grained…
How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models
Abdelrahman Abdallah, Bhawna Piryani, Jamshid Mozafari +2
In this work, we present a systematic and comprehensive empirical evaluation of state-of-the-art reranking methods, encompassing large language model (LLM)-based, lightweight conte…
ClaimGen-CN: A Large-scale Chinese Dataset for Legal Claim Generation
Siying Zhou, Yiquan Wu, Hui Chen +6
Legal claims refer to the plaintiff's demands in a case and are essential to guiding judicial reasoning and case resolution. While many works have focused on improving the efficien…
Evaluating List Construction and Temporal Understanding capabilities of Large Language Models
Alexandru Dumitru, V Venktesh, Adam Jatowt +1
Large Language Models (LLMs) have demonstrated immense advances in a wide range of natural language tasks. However, these models are susceptible to hallucinations and errors on par…
It's High Time: A Survey of Temporal Question Answering
Bhawna Piryani, Abdelrahman Abdallah, Jamshid Mozafari +2
Time plays a critical role in how information is generated, retrieved, and interpreted. In this survey, we provide a comprehensive overview of Temporal Question Answering (TQA), a…
A Study into Investigating Temporal Robustness of LLMs
Jonas Wallat, Abdelrahman Abdallah, Adam Jatowt +1
Large Language Models (LLMs) encapsulate a surprising amount of factual world knowledge. However, their performance on temporal questions and historical knowledge is limited becaus…
Evaluating Answer Reranking Strategies in Time-sensitive Question Answering
Mehmet Kardan, Bhawna Piryani, Adam Jatowt
Despite advancements in state-of-the-art models and information retrieval techniques, current systems still struggle to handle temporal information and to correctly answer detailed…
From Retrieval to Generation: Comparing Different Approaches
Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani +2
Knowledge-intensive tasks, particularly open-domain question answering (ODQA), document reranking, and retrieval-augmented language modeling, require a balance between retrieval ac…
Evaluating Robustness of LLMs in Question Answering on Multilingual Noisy OCR Data
Bhawna Piryani, Jamshid Mozafari, Abdelrahman Abdallah +2
Optical Character Recognition (OCR) plays a crucial role in digitizing historical and multilingual documents, yet OCR errors - imperfect extraction of text, including character ins…
Wrong Answers Can Also Be Useful: PlausibleQA -- A Large-Scale QA Dataset with Answer Plausibility Scores
Jamshid Mozafari, Abdelrahman Abdallah, Bhawna Piryani +1
Large Language Models (LLMs) are revolutionizing information retrieval, with chatbots becoming an important source for answering user queries. As by their design, LLMs prioritize g…
HintEval: An Open-Source Python Toolkit for Hint Generation and Hint Evaluation
Jamshid Mozafari, Bhawna Piryani, Abdelrahman Abdallah +1
Large Language Models (LLMs) increasingly provide direct answers to user questions, raising concerns about reduced engagement in critical thinking and problem-solving. Hint generat…
ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval
Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani +1
Retrieval-Augmented Generation (RAG) models have drawn considerable attention in modern open-domain question answering. The effectiveness of RAG depends on the quality of the top r…
Navigating Tomorrow: Reliably Assessing Large Language Models Performance on Future Event Prediction
Petraq Nako, Adam Jatowt
Predicting future events is an important activity with applications across multiple fields and domains. For example, the capacity to foresee stock market trends, natural disasters,…
WikiHint: A Human-Annotated Dataset for Hint Ranking and Generation
Jamshid Mozafari, Florian Gerhold, Adam Jatowt
The use of Large Language Models (LLMs) has increased significantly with users frequently asking questions to chatbots. In the time when information is readily accessible, it is cr…