Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
arXiv:2110.08455
Abstract
Pretrained Language Models (PLM) have established a new paradigm through learning informative contextualized representations on large-scale text corpus. This new paradigm has revolutionized the entire field of natural language processing, and set the new state-of-the-art performance for a wide variety of NLP tasks. However, though PLMs could store certain knowledge/facts from training corpus, their knowledge awareness is still far from satisfactory. To address this issue, integrating knowledge into PLMs have recently become a very active research area and a variety of approaches have been developed. In this paper, we provide a comprehensive survey of the literature on this emerging and fast-growing field - Knowledge Enhanced Pretrained Language Models (KE-PLMs). We introduce three taxonomies to categorize existing work. Besides, we also survey the various NLU and NLG applications on which KE-PLM has demonstrated superior performance over vanilla PLMs. Finally, we discuss challenges that face KE-PLMs and also promising directions for future research.
References in corpus (7)
- Distilling the Knowledge in a Neural Network
- Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
- The Effect of Natural Distribution Shift on Question Answering Models
- Combining pre-trained language models and structured knowledge
- KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
- Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection
- Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees