Why Do Masked Neural Language Models Still Need Common Sense Knowledge?

Abstract

Currently, contextualized word representations are learned by intricate neural network models, such as masked neural language models (MNLMs). The new representations significantly enhanced the performance in automated question answering by reading paragraphs. However, identifying the detailed knowledge trained in the MNLMs is difficult owing to numerous and intermingled parameters. This paper provides empirical but insightful analyses on the pretrained MNLMs with respect to common sense knowledge. First, we propose a test that measures what types of common sense knowledge do pretrained MNLMs understand. From the test, we observed that MNLMs partially understand various types of common sense knowledge but do not accurately understand the semantic meaning of relations. In addition, based on the difficulty of the question-answering task problems, we observed that pretrained MLM-based models are still vulnerable to problems that require common sense knowledge. We also experimentally demonstrated that we can elevate existing MNLM-based models by combining knowledge from an external common sense repository.

References in corpus (6)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
• What do you learn from context? Probing for sentence structure in contextualized word representations
• Dynamic Integration of Background Knowledge in Neural NLU Systems
• Commonsense Knowledge Mining from Pretrained Models

Cited by in corpus (3)

• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
• Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks
• ExBERT: An External Knowledge Enhanced BERT for Natural Language Inference