7 papers
Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP
Urja Khurana, Michiel van der Meer, Enrico Liscio +2
Subjective judgments are part of several NLP datasets and recent work is increasingly prioritizing models whose outputs reflect this diversity of perspectives. Such responses allow…
DefVerify: Do Hate Speech Models Reflect Their Dataset's Definition?
Urja Khurana, Eric Nalisnick, Antske Fokkens
When building a predictive model, it is often difficult to ensure that application-specific requirements are encoded by the model that will eventually be deployed. Consider researc…
Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?
Urja Khurana, Eric Nalisnick, Antske Fokkens +1
Subjective tasks in NLP have been mostly relegated to objective standards, where the gold label is decided by taking the majority vote. This obfuscates annotator disagreement and t…
Leveraging Few-Shot Data Augmentation and Waterfall Prompting for Response Generation
Lea Krause, Selene Báez Santamaría, Michiel van der Meer +1
This paper discusses our approaches for task-oriented conversational modelling using subjective knowledge, with a particular emphasis on response generation. Our methodology was sh…
Will It Blend? Mixing Training Paradigms & Prompting for Argument Quality Prediction
Michiel van der Meer, Myrthe Reuver, Urja Khurana +2
This paper describes our contributions to the Shared Task of the 9th Workshop on Argument Mining (2022). Our approach uses Large Language Models for the task of Argument Quality Pr…
Hate Speech Criteria: A Modular Approach to Task-Specific Hate Speech Definitions
Urja Khurana, Ivar Vermeulen, Eric Nalisnick +2
\textbf{Offensive Content Warning}: This paper contains offensive language only for providing examples that clarify this research and do not reflect the authors' opinions. Please b…
How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task
Urja Khurana, Eric Nalisnick, Antske Fokkens
Despite their success, modern language models are fragile. Even small changes in their training pipeline can lead to unexpected results. We study this phenomenon by examining the r…