Go Beyond Plain Fine-tuning: Improving Pretrained Models for Social Commonsense
arXiv:2105.05913
Abstract
Pretrained language models have demonstrated outstanding performance in many NLP tasks recently. However, their social intelligence, which requires commonsense reasoning about the current situation and mental states of others, is still developing. Towards improving language models' social intelligence, we focus on the Social IQA dataset, a task requiring social and emotional commonsense reasoning. Building on top of the pretrained RoBERTa and GPT2 models, we propose several architecture variations and extensions, as well as leveraging external commonsense corpora, to optimize the model for Social IQA. Our proposed system achieves competitive results as those top-ranking models on the leaderboard. This work demonstrates the strengths of pretrained language models, and provides viable ways to improve their performance for a particular task.
arXiv commentSLT 2021
References in corpus (7)
- Overcoming catastrophic forgetting in neural networks
- Bidirectional Attention Flow for Machine Comprehension
- Pathologies of Neural Models Make Interpretations Difficult
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
- Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?
- How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG
- How Additional Knowledge can Improve Natural Language Commonsense Question Answering?