Relational World Knowledge Representation in Contextual Language Models: A Review
arXiv:2104.05837
Abstract
Relational knowledge bases (KBs) are commonly used to represent world knowledge in machines. However, while advantageous for their high degree of precision and interpretability, KBs are usually organized according to manually-defined schemas, which limit their expressiveness and require significant human efforts to engineer and maintain. In this review, we take a natural language processing perspective to these limitations, examining how they may be addressed in part by training deep contextual language models (LMs) to internalize and express relational knowledge in more flexible forms. We propose to organize knowledge representation strategies in LMs by the level of KB supervision provided, from no KB supervision at all to entity- and relation-level supervision. Our contributions are threefold: (1) We provide a high-level, extensible taxonomy for knowledge representation in LMs; (2) Within our taxonomy, we highlight notable models, evaluation tasks, and findings, in order to provide an up-to-date review of current knowledge representation capabilities in LMs; and (3) We suggest future research directions that build upon the complementary aspects of LMs and KBs as knowledge representations.
arXiv commentEMNLP 2021
References in corpus (15)
- Language Models are Few-Shot Learners
- ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
- KG-BERT: BERT for Knowledge Graph Completion
- Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
- True Few-Shot Learning with Language Models
- Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases
- Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking
- How Context Affects Language Models' Factual Predictions
- Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
- Knowledge-Aware Language Model Pretraining
- Measuring and Improving Consistency in Pretrained Language Models
- Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
- Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge
- ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
- Combining pre-trained language models and structured knowledge