Attention-based Contrastive Learning for Winograd Schemas
arXiv:2109.05108
Abstract
Self-supervised learning has recently attracted considerable attention in the NLP community for its ability to learn discriminative features using a contrastive objective. This paper investigates whether contrastive learning can be extended to Transfomer attention to tackling the Winograd Schema Challenge. To this end, we propose a novel self-supervised framework, leveraging a contrastive loss directly at the level of self-attention. Experimental analysis of our attention-based models on multiple datasets demonstrates superior commonsense reasoning capabilities. The proposed approach outperforms all comparable unsupervised approaches while occasionally surpassing supervised ones.
arXiv commentTo appear at EMNLP 2021 (findings)
References in corpus (9)
- A Simple Method for Commonsense Reasoning
- The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence
- A Surprisingly Robust Trick for Winograd Schema Challenge
- Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
- How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG
- CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
- A Review of Winograd Schema Challenge Datasets and Approaches
- Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge
- Do Fine-tuned Commonsense Language Models Really Generalize?