Combining pre-trained language models and structured knowledge
arXiv:2101.12294
Abstract
In recent years, transformer-based language models have achieved state of the art performance in various NLP benchmarks. These models are able to extract mostly distributional information with some semantics from unstructured text, however it has proven challenging to integrate structured information, such as knowledge graphs into these models. We examine a variety of approaches to integrate structured knowledge into current language models and determine challenges, and possible opportunities to leverage both structured and unstructured information sources. From our survey, we find that there are still opportunities at exploiting adapter-based injections and that it may be possible to further combine various of the explored approaches into one system.
Updated references
References in corpus (14)
- Overcoming catastrophic forgetting in neural networks
- ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
- KG-BERT: BERT for Knowledge Graph Completion
- Parameter-Efficient Transfer Learning for NLP
- K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
- Language Models as Knowledge Bases?
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
- Knowledge-Aware Language Model Pretraining
- Commonsense Knowledge Mining from Pretrained Models
- Integrating Graph Contextualized Knowledge into Pre-trained Language Models
- Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations
- Knowledge Enhanced Contextual Word Representations
- Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning