Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
arXiv:2110.08455
Abstract
Pretrained Language Models (PLM) have established a new paradigm through learning informative contextualized representations on large-scale text corpus. This new paradigm has revolutionized the entire field of natural language processing, and set the new state-of-the-art performance for a wide variety of NLP tasks. However, though PLMs could store certain knowledge/facts from training corpus, their knowledge awareness is still far from satisfactory. To address this issue, integrating knowledge into PLMs have recently become a very active research area and a variety of approaches have been developed. In this paper, we provide a comprehensive survey of the literature on this emerging and fast-growing field - Knowledge Enhanced Pretrained Language Models (KE-PLMs). We introduce three taxonomies to categorize existing work. Besides, we also survey the various NLU and NLG applications on which KE-PLM has demonstrated superior performance over vanilla PLMs. Finally, we discuss challenges that face KE-PLMs and also promising directions for future research.
References in corpus (26)
- Distilling the Knowledge in a Neural Network
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Pre-trained Models for Natural Language Processing: A Survey
- ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
- KG-BERT: BERT for Knowledge Graph Completion
- Universal Transformers
- SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Strategies for Pre-training Graph Neural Networks
- K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
- Language Models are Open Knowledge Graphs
- KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
- Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
- Knowledge-Aware Language Model Pretraining
- Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
- The Effect of Natural Distribution Shift on Question Answering Models
- E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce
- KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning
- Augmenting Transformers with KNN-Based Composite Memory for Dialogue
- ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
- Combining pre-trained language models and structured knowledge
- KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
- Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection
- Latent Relation Language Models
- Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees
- Relational World Knowledge Representation in Contextual Language Models: A Review