Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection

Abstract

Stance Detection is concerned with identifying the attitudes expressed by an author towards a target of interest. This task spans a variety of domains ranging from social media opinion identification to detecting the stance for a legal claim. However, the framing of the task varies within these domains, in terms of the data collection protocol, the label dictionary and the number of available annotations. Furthermore, these stance annotations are significantly imbalanced on a per-topic and inter-topic basis. These make multi-domain stance detection a challenging task, requiring standardization and domain adaptation. To overcome this challenge, we propose $\text{T}$opic $\text{E}$fficient $\text{St}$anc$\text{E}$ $\text{D}$etection (TESTED), consisting of a topic-guided diversity sampling technique and a contrastive objective that is used for fine-tuning a stance classifier. We evaluate the method on an existing benchmark of $16$ datasets with in-domain, i.e. all topics seen and out-of-domain, i.e. unseen topics, experiments. The results show that our method outperforms the state-of-the-art with an average of $3.5$ F1 points increase in-domain, and is more generalizable with an averaged increase of $10.2$ F1 on out-of-domain evaluation while using $\le 10\%$ of the training data. We show that our sampling technique mitigates both inter- and per-topic class imbalances. Finally, our analysis demonstrates that the contrastive learning objective allows the model a more pronounced segmentation of samples with varying labels.

arXiv commentACL 2023 (Oral)

References in corpus (10)

• Decoupled Weight Decay Regularization
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
• True Few-Shot Learning with Language Models
• Get out the vote: Determining support or opposition from Congressional floor-debate transcripts
• Discourse-Aware Rumour Stance Classification in Social Media Using Sequential Classifiers
• The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants
• Contrastive Learning for Image Captioning
• Active Learning: Problem Settings and Recent Developments
• Mitigating shortage of labeled data using clustering-based active learning with diversity exploration

Cited by in corpus (1)

• Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI