11 papers
Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation
Sadia Kamal, Arefa Patwary, Anthony Marchiafava +2
Survey-style evaluations of large language models often treat a prompted response as a measure of a model's values or beliefs. This assumption is particularly fragile when response…
Evaluation Framework for Highlight Explanations of Context Utilisation in Language Models
Jingyi Sun, Pepa Atanasova, Sagnik Ray Choudhury +2
Context utilisation, the ability of Language Models (LMs) to incorporate relevant information from the provided context when generating responses, remains largely opaque to users,…
S-GRADES -- Studying Generalization of Student Response Assessments in Diverse Evaluative Settings
Tasfia Seuti, Sagnik Ray Choudhury
Evaluating student responses, from long essays to short factual answers, is a key challenge in educational NLP. Automated Essay Scoring (AES) focuses on holistic writing qualities…
LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering
Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray Choudhury
Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-H…
NLP for Social Good: A Survey and Outlook of Challenges, Opportunities, and Responsible Deployment
Antonia Karamolegkou, Angana Borah, Eunjung Cho +30
Natural language processing (NLP) now shapes many aspects of our world, yet its potential for positive social impact is underexplored. This paper surveys work in ``NLP for Social G…
A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models
Sadia Kamal, Lalu Prasad Yadav Prakash, S M Rafiuddin +3
The Political Compass Test (PCT) and similar surveys are commonly used to assess political bias in auto-regressive LLMs. Our rigorous statistical experiments show that while change…