32 citations · 202 across the 82 of their papers we have counts for
70 papers · 1 filter
OLMo-Detect: A Multi-Stage, Confounder-Controlled Benchmark for Membership Inference on Large Language Models
Tao Shi, Chaoyi Xiang, Qiongkai Xu +1
Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus. De…
Enhancing Assessment of Self-Consistency in LLM Explanations using Perturbation Strength
Phuong Q. Le, Kemal Kurniawan, Jey Han Lau
Prior work has examined the self-consistency of LLM-generated explanations using surface-level perturbation methods. However, the strength of these perturbations is not explicitly…
AutoLexSteer: Automatic Contrast Construction for Lexical Activation Steering
Shuhe Wang, Lachlan Cowley, Eduard Hovy +1
Steering vectors have rapidly emerged as a popular and effective method for guiding the output of LLMs in very specific ways. But constructing accurate steering vectors is a diffic…
MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shifts
Matteo Greco, Anudeex Shetty, Andrea Tagarelli +1
While existing work on LLM authorship attribution (AA) has made progress, available benchmarks remain limited, often focusing on English, controlled settings, or relatively outdate…
CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics
Ming-Bin Chen, Jey Han Lau, Lea Frermann
Measuring the quality of public deliberation requires evaluating not only civility or argument structure, but also the informational progress of a conversation. We introduce a fram…
Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning
Yanbei Jiang, Amr Keleg, Ryandito Diandaru +4
While the real world is inherently stochastic, Large Language Models (LLMs) are predominantly evaluated on single-round inference against fixed ground truths. In this work, we shif…
Inclusion-of-Thoughts: Mitigating Preference Instability via Purifying the Decision Space
Mohammad Reza Ghasemi Madani, Soyeon Caren Han, Shuo Yang +1
Multiple-choice questions (MCQs) are widely used to evaluate large language models (LLMs). However, LLMs remain vulnerable to the presence of plausible distractors. This often dive…
Who Wrote the Book? Detecting and Attributing LLM Ghostwriters
Anudeex Shetty, Qiongkai Xu, Olga Ohrimenko +1
In this paper, we introduce GhostWriteBench, a dataset for LLM authorship attribution. It comprises long-form texts (50K+ words per book) generated by frontier LLMs, and is designe…
Context Volume Drives Performance: Tackling Domain Shift in Extremely Low-Resource Translation via RAG
David Samuel Setiawan, Raphaël Merx, Jey Han Lau
Neural Machine Translation (NMT) models for low-resource languages suffer significant performance degradation under domain shift. We quantify this challenge using Dhao, an indigeno…
COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations
Rui Xing, Preslav Nakov, Timothy Baldwin +1
Fact-checking on major platforms, such as X, Meta, and TikTok, is shifting from expert-driven verification to a community-based setup, where users contribute explanatory notes to c…
On the Interplay between Human Label Variation and Model Fairness
Kemal Kurniawan, Meladel Mistica, Timothy Baldwin +1
The impact of human label variation (HLV) on model fairness is an unexplored topic. This paper examines the interplay by comparing training on majority-vote labels with a range of…
FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation
Yulia Otmakhova, Hung Thinh Truong, Rahmad Mahendra +4
We present FLUKE (Framework for LingUistically-driven and tasK-agnostic robustness Evaluation), a framework for assessing model robustness through systematic minimal variations of…
Moderation Matters:Measuring Conversational Moderation Impact in English as a Second Language Group Discussion
Rena Gao, Ming-Bin Chen, Lea Frermann +1
English as a Second Language (ESL) speakers often struggle to engage in group discussions due to language barriers. While moderators can facilitate participation, few studies asses…
Can LLMs Simulate L2-English Dialogue? An Information-Theoretic Analysis of L1-Dependent Biases
Rena Gao, Xuetong Wu, Tatsuki Kuribayashi +6
This study evaluates Large Language Models' (LLMs) ability to simulate non-native-like English use observed in human second language (L2) learners interfered with by their native f…
Beyond Seen Data: Improving KBQA Generalization Through Schema-Guided Logical Form Generation
Shengxiang Gao, Jey Han Lau, Jianzhong Qi
Knowledge base question answering (KBQA) aims to answer user questions in natural language using rich human knowledge stored in large KBs. As current KBQA methods struggle with uns…
Decomposed Opinion Summarization with Verified Aspect-Aware Modules
Miao Li, Jey Han Lau, Eduard Hovy +1
Opinion summarization plays a key role in deriving meaningful insights from large-scale online reviews. To make the process more explainable and grounded, we propose a domain-agnos…
WHoW: A Cross-domain Approach for Analysing Conversation Moderation
Ming-Bin Chen, Lea Frermann, Jey Han Lau
We propose WHoW, an evaluation framework for analyzing the facilitation strategies of moderators across different domains/scenarios by examining their motives (Why), dialogue acts…
MoDEM: Mixture of Domain Expert Models
Toby Simonds, Kemal Kurniawan, Jey Han Lau
We propose a novel approach to enhancing the performance and efficiency of large language models (LLMs) by combining domain prompt routing with domain-specialized models. We introd…
To Aggregate or Not to Aggregate. That is the Question: A Case Study on Annotation Subjectivity in Span Prediction
Kemal Kurniawan, Meladel Mistica, Timothy Baldwin +1
This paper explores the task of automatic prediction of text spans in a legal problem description that support a legal area label. We use a corpus of problem descriptions written b…
An Interpretable and Crosslingual Method for Evaluating Second-Language Dialogues
Rena Gao, Jingxuan Wu, Xuetong Wu +4
We analyse the cross-lingual transferability of a dialogue evaluation framework that assesses the relationships between micro-level linguistic features (e.g. backchannels) and macr…
Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversations
Rena Gao, Carsten Roever, Jey Han Lau
We present an evaluation framework for interactive dialogue assessment in the context of English as a Second Language (ESL) speakers. Our framework collects dialogue-level interact…
Factual Dialogue Summarization via Learning from Large Language Models
Rongxin Zhu, Jey Han Lau, Jianzhong Qi
Factual consistency is an important quality in dialogue summarization. Large language model (LLM)-based automatic text summarization models generate more factually consistent summa…
Evaluating Evidence Attribution in Generated Fact Checking Explanations
Rui Xing, Timothy Baldwin, Jey Han Lau
Automated fact-checking systems often struggle with trustworthiness, as their generated explanations can include hallucinations. In this work, we explore evidence attribution for f…
CMA-R:Causal Mediation Analysis for Explaining Rumour Detection
Lin Tian, Xiuzhen Zhang, Jey Han Lau
We apply causal mediation analysis to explain the decision-making process of neural models for rumour detection on Twitter. Interventions at the input and network level reveal the…