Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models

Abstract

The state-of-the-art pre-trained language representation models, such as Bidirectional Encoder Representations from Transformers (BERT), rarely incorporate commonsense knowledge or other knowledge explicitly. We propose a pre-training approach for incorporating commonsense knowledge into language representation models. We construct a commonsense-related multi-choice question answering dataset for pre-training a neural language representation model. The dataset is created automatically by our proposed "align, mask, and select" (AMS) method. We also investigate different pre-training tasks. Experimental results demonstrate that pre-training models using the proposed approach followed by fine-tuning achieve significant improvements over previous state-of-the-art models on two commonsense-related benchmarks, including CommonsenseQA and Winograd Schema Challenge. We also observe that fine-tuned models after the proposed pre-training approach maintain comparable performance on other NLP tasks, such as sentence classification and natural language inference tasks, compared to the original BERT models. These results verify that the proposed approach, while significantly improving commonsense-related NLP tasks, does not degrade the general language representation capabilities.

References in corpus (12)

• Distributed Representations of Words and Phrases and their Compositionality
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
• ERNIE: Enhanced Representation through Knowledge Integration
• A large annotated corpus for learning natural language inference
• A Simple Method for Commonsense Reasoning
• ERNIE: Enhanced Language Representation with Informative Entities
• A Surprisingly Robust Trick for Winograd Schema Challenge
• How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG
• Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge
• Improving Question Answering by Commonsense-Based Pre-Training
• A Knowledge Hunting Framework for Common Sense Reasoning

Cited by in corpus (23)

• QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
• ALBERT with Knowledge Graph Encoder Utilizing Semantic Similarity for Commonsense Question Answering
• Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering
• A Review of Winograd Schema Challenge Datasets and Approaches
• Teaching Pretrained Models with Commonsense Reasoning: A Preliminary KB-Based Approach
• The Life Cycle of Knowledge in Big Language Models: A Survey
• Combining pre-trained language models and structured knowledge
• KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
• Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
• KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
• Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph
• Zero-shot Visual Question Answering using Knowledge Graph
• Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
• Relational World Knowledge Representation in Contextual Language Models: A Review
• On Commonsense Cues in BERT for Solving Commonsense Tasks
• Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction
• Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks
• Attention-based Contrastive Learning for Winograd Schemas
• Beyond Language: Learning Commonsense from Images for Reasoning