331 citations · 334 across the 10 of their papers we have counts for
12 papers
Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews
Ehsan Barkhordar, Abdulfattah Safa, Verena Blaschke +3
Peer review plays a central role in the NLP publication process, but is susceptible to various biases. Here, we study language-of-study (LoS) bias: the tendency for reviewers to ev…
Humans and LLMs Diverge on Probabilistic Inferences
Gaurav Kamath, Sreenath Madathil, Sebastian Schuster +2
Human reasoning often involves working over limited information to arrive at probabilistic conclusions. In its simplest form, this involves making an inference that is not strictly…
Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations
Pingjun Hong, Beiduo Chen, Siyao Peng +3
Natural Language Inference (NLI) datasets often exhibit human label variation. To better understand these variations, explanation-based approaches analyze the underlying reasoning…
A survey of diversity quantification in natural language processing: The why, what, where and how
Louis Estève, Marie-Catherine de Marneffe, Nurit Melnik +2
The concept of diversity has received increasing attention in natural language processing (NLP) in recent years. It became an advocated property of datasets and systems, and many m…
LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference
Pingjun Hong, Beiduo Chen, Siyao Peng +2
There is increasing evidence of Human Label Variation (HLV) in Natural Language Inference (NLI), where annotators assign different labels to the same premise-hypothesis pair. Howev…
Explanation sensitivity to the randomness of large language models: the case of journalistic text classification
Jeremie Bogaert, Marie-Catherine de Marneffe, Antonin Descampe +3
Large language models (LLMs) perform very well in several natural language processing tasks but raise explainability challenges. In this paper, we examine the effect of random elem…
VariErr NLI: Separating Annotation Error from Human Label Variation
Leon Weber-Genzel, Siyao Peng, Marie-Catherine de Marneffe +1
Human label variation arises when annotators assign different labels to the same item for valid reasons, while annotation errors occur when labels are assigned for invalid reasons.…
Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation
Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe
Human label variation (Plank 2022), or annotation disagreement, exists in many natural language processing (NLP) tasks. To be robust and trusted, NLP models need to identify such v…
Investigating Reasons for Disagreement in Natural Language Inference
Nan-Jiang Jiang, Marie-Catherine de Marneffe
We investigate how disagreement in natural language inference (NLI) annotation arises. We developed a taxonomy of disagreement sources with 10 categories spanning 3 high-level clas…
He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics
Nanjiang Jiang, Marie-Catherine de Marneffe
We investigate how well BERT performs on predicting factuality in several existing English datasets, encompassing various linguistic constructions. Although BERT obtains a strong p…
Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter +6
Universal Dependencies is an open community effort to create cross-linguistically consistent treebank annotation for many languages within a dependency-based lexicalist framework.…
"i have a feeling trump will win..................": Forecasting Winners and Losers from User Predictions on Twitter
Sandesh Swamy, Alan Ritter, Marie-Catherine de Marneffe
Social media users often make explicit predictions about upcoming events. Such statements vary in the degree of certainty the author expresses toward the outcome:"Leonardo DiCaprio…