Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge

Abstract

Winograd Schema Challenge (WSC) was proposed as an AI-hard problem in testing computers' intelligence on common sense representation and reasoning. This paper presents the new state-of-theart on WSC, achieving an accuracy of 71.1%. We demonstrate that the leading performance benefits from jointly modelling sentence structures, utilizing knowledge learned from cutting-edge pretraining models, and performing fine-tuning. We conduct detailed analyses, showing that fine-tuning is critical for achieving the performance, but it helps more on the simpler associative problems. Modelling sentence dependency structures, however, consistently helps on the harder non-associative subset of WSC. Analysis also shows that larger fine-tuning datasets yield better performances, suggesting the potential benefit of future work on annotating more Winograd schema sentences.

7 pages

References in corpus (4)

• Distributed Representations of Words and Phrases and their Compositionality
• Playing Atari with Deep Reinforcement Learning
• A Simple Method for Commonsense Reasoning
• How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG

Cited by in corpus (7)

• Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
• How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG
• A Review of Winograd Schema Challenge Datasets and Approaches
• Can a Gorilla Ride a Camel? Learning Semantic Plausibility from Text
• Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction
• Attention-based Contrastive Learning for Winograd Schemas
• Precise Task Formalization Matters in Winograd Schema Evaluations