A Surprisingly Robust Trick for Winograd Schema Challenge

Abstract

The Winograd Schema Challenge (WSC) dataset WSC273 and its inference counterpart WNLI are popular benchmarks for natural language understanding and commonsense reasoning. In this paper, we show that the performance of three language models on WSC273 strongly improves when fine-tuned on a similar pronoun disambiguation problem dataset (denoted WSCR). We additionally generate a large unsupervised WSC-like dataset. By fine-tuning the BERT language model both on the introduced and on the WSCR dataset, we achieve overall accuracies of 72.5% and 74.7% on WSC273 and WNLI, improving the previous state-of-the-art solutions by 8.8% and 9.6%, respectively. Furthermore, our fine-tuned models are also consistently more robust on the "complex" subsets of WSC273, introduced by Trichelair et al. (2018).

arXiv commentAppeared as part of the ACL 2019 conference

References in corpus (6)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Achieving Human Parity on Automatic Chinese to English News Translation
• A Simple Method for Commonsense Reasoning
• Multi-Task Deep Neural Networks for Natural Language Understanding
• How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG
• A Knowledge Hunting Framework for Common Sense Reasoning

Cited by in corpus (27)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• True Few-Shot Learning with Language Models
• Towards Natural Language Interfaces for Data Visualization: A Survey
• WinoGrande: An Adversarial Winograd Schema Challenge at Scale
• Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Explaining Deep Neural Networks
• A Review of Winograd Schema Challenge Datasets and Approaches
• Coreferential Reasoning Learning for Language Representation
• Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
• Robust Encodings: A Framework for Combating Adversarial Typos
• A Hybrid Neural Network Model for Commonsense Reasoning
• ASER: A Large-scale Eventuality Knowledge Graph
• A Brief Survey and Comparative Study of Recent Development of Pronoun Coreference Resolution
• WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge
• A Semantic-based Method for Unsupervised Commonsense Question Answering
• From Discourse to Narrative: Knowledge Projection for Event Relation Extraction
• Bridging Anaphora Resolution as Question Answering
• OCHADAI-KYOTO at SemEval-2021 Task 1: Enhancing Model Generalization and Robustness for Lexical Complexity Prediction
• UU-Tax at SemEval-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentation
• Attention-based Contrastive Learning for Winograd Schemas
• Experience and Prediction: A Metric of Hardness for a Novel Litmus Test
• Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction
• Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models
• Precise Task Formalization Matters in Winograd Schema Evaluations