Commonsense Knowledge Mining from Pretrained Models

Abstract

Inferring commonsense knowledge is a key challenge in natural language processing, but due to the sparsity of training data, previous work has shown that supervised methods for commonsense knowledge mining underperform when evaluated on novel data. In this work, we develop a method for generating commonsense knowledge using a large, pre-trained bidirectional language model. By transforming relational triples into masked sentences, we can use this model to rank a triple's validity by the estimated pointwise mutual information between the two entities. Since we do not update the weights of the bidirectional model, our approach is not biased by the coverage of any one commonsense knowledge base. Though this method performs worse on a test set than models explicitly trained on a corresponding training set, it outperforms these methods when mining commonsense knowledge from new sources, suggesting that unsupervised techniques may generalize better than current supervised approaches.

References in corpus (2)

• A Simple Method for Commonsense Reasoning
• A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories

Cited by in corpus (26)

• Pre-trained Models for Natural Language Processing: A Survey
• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Making Pre-trained Language Models Better Few-shot Learners
• Entailment as Few-Shot Learner
• SentiPrompt: Sentiment Knowledge Enhanced Prompt-Tuning for Aspect-Based Sentiment Analysis
• Measuring and Improving Consistency in Pretrained Language Models
• Improving BERT Performance for Aspect-Based Sentiment Analysis
• Pre-Trained Models: Past, Present and Future
• Why Do Masked Neural Language Models Still Need Common Sense Knowledge?
• Combining pre-trained language models and structured knowledge
• Learning to Deceive Knowledge Graph Augmented Models via Targeted Perturbation
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
• On the Role of Conceptualization in Commonsense Knowledge Graph Construction
• Enriching a Model's Notion of Belief using a Persistent Memory
• TransOMCS: From Linguistic Graphs to Commonsense Knowledge
• BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?
• Towards a Universal Continuous Knowledge Base
• Probing Pre-Trained Language Models for Disease Knowledge
• Commonsense Knowledge Mining from Term Definitions
• SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning
• P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts
• Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings
• Few-shot Named Entity Recognition with Cloze Questions
• Alleviating the Knowledge-Language Inconsistency: A Study for Deep Commonsense Knowledge
• IIE-NLP-NUT at SemEval-2020 Task 4: Guiding PLM with Prompt Template Reconstruction Strategy for ComVE