A Simple Method for Commonsense Reasoning

Abstract

Commonsense reasoning is a long-standing challenge for deep learning. For example, it is difficult to use neural networks to tackle the Winograd Schema dataset (Levesque et al., 2011). In this paper, we present a simple method for commonsense reasoning with neural networks, using unsupervised learning. Key to our method is the use of language models, trained on a massive amount of unlabled data, to score multiple choice questions posed by commonsense reasoning tests. On both Pronoun Disambiguation and Winograd Schema challenges, our models outperform previous state-of-the-art methods by a large margin, without using expensive annotated knowledge bases or hand-engineered features. We train an array of large RNN language models that operate at word or character level on LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and a customized corpus for this task and show that diversity of training data plays an important role in test performance. Further analysis also shows that our system successfully discovers important features of the context that decide the correct answer, indicating a good grasp of commonsense knowledge.

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• Broad Context Language Modeling as Reading Comprehension

Cited by in corpus (102)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• MPNet: Masked and Permuted Pre-training for Language Understanding
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
• Big Bird: Transformers for Longer Sequences
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Making Pre-trained Language Models Better Few-shot Learners
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• Generative Data Augmentation for Commonsense Reasoning
• A Surprisingly Robust Trick for Winograd Schema Challenge
• Adversarial Training for Large Neural Language Models
• WinoGrande: An Adversarial Winograd Schema Challenge at Scale
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
• Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting
• Luna: Linear Unified Nested Attention
• Explaining Question Answering Models through Text Generation
• Commonsense Knowledge Mining from Pretrained Models
• How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG
• Identifying Machine-Paraphrased Plagiarism
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Modern Baselines for SPARQL Semantic Parsing
• Large language models as oracles for instantiating ontologies with domain-specific knowledge
• KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• PTR: Prompt Tuning with Rules for Text Classification
• Birds of a Feather Flock Together: Satirical News Detection via Language Model Differentiation
• Neural Language Generation: Formulation, Methods, and Evaluation
• A Review of Winograd Schema Challenge Datasets and Approaches
• One Epoch Is All You Need
• Evaluating Biased Attitude Associations of Language Models in an Intersectional Context
• Coreferential Reasoning Learning for Language Representation
• Teaching Pretrained Models with Commonsense Reasoning: A Preliminary KB-Based Approach
• MatSciRE: Leveraging Pointer Networks to Automate Entity and Relation Extraction for Material Science Knowledge-base Construction
• The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
• Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge
• Framing the News:From Human Perception to Large Language Model Inferences
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
• Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models
• MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models
• BioMegatron: Larger Biomedical Domain Language Model
• Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Function
• Transfer Learning from Transformers to Fake News Challenge Stance Detection (FNC-1) Task
• On the comparability of Pre-trained Language Models
• ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
• Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization
• ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding
• Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns
• On the Role of Conceptualization in Commonsense Knowledge Graph Construction
• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
• Automatic Story Generation: Challenges and Attempts
• A Hybrid Neural Network Model for Commonsense Reasoning
• Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text Classification
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining
• Blacks is to Anger as Whites is to Joy? Understanding Latent Affective Bias in Large Pre-trained Neural Language Models
• Structural Similarities Between Language Models and Neural Response Measurements
• Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph
• ASER: A Large-scale Eventuality Knowledge Graph
• Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning
• WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge
• How Vulnerable Are Automatic Fake News Detection Methods to Adversarial Attacks?
• A Brief Survey and Comparative Study of Recent Development of Pronoun Coreference Resolution
• Learning to Explain: Answering Why-Questions via Rephrasing
• A Knowledge Hunting Framework for Common Sense Reasoning
• 8-bit Optimizers via Block-wise Quantization
• Narrative Incoherence Detection
• Probing Across Time: What Does RoBERTa Know and When?
• From Discourse to Narrative: Knowledge Projection for Event Relation Extraction
• Commonsense knowledge adversarial dataset that challenges ELECTRA
• Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
• Unsupervised Deep Structured Semantic Models for Commonsense Reasoning
• Multiplicative Position-aware Transformer Models for Language Understanding
• P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts
• Neural Search: Learning Query and Product Representations in Fashion E-commerce
• Precise Task Formalization Matters in Winograd Schema Evaluations
• Towards an Atlas of Cultural Commonsense for Machine Reasoning
• On Commonsense Cues in BERT for Solving Commonsense Tasks
• Open Rule Induction
• Social Networks Analysis to Retrieve Critical Comments on Online Platforms
• Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction
• Evaluating Document Coherence Modelling
• CALM: Continuous Adaptive Learning for Language Modeling
• Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you?
• XD at SemEval-2020 Task 12: Ensemble Approach to Offensive Language Identification in Social Media Using Transformer Encoders
• QiaoNing at SemEval-2020 Task 4: Commonsense Validation and Explanation system based on ensemble of language model
• Alleviating the Knowledge-Language Inconsistency: A Study for Deep Commonsense Knowledge
• EntEval: A Holistic Evaluation Benchmark for Entity Representations
• Experience and Prediction: A Metric of Hardness for a Novel Litmus Test
• The Knowref Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution
• Attention-based Contrastive Learning for Winograd Schemas