A Survey of Knowledge Enhanced Pre-trained Models
arXiv:2110.00269
Abstract
Pre-trained language models learn informative word representations on a large-scale text corpus through self-supervised learning, which has achieved promising performance in fields of natural language processing (NLP) after fine-tuning. These models, however, suffer from poor robustness and lack of interpretability. We refer to pre-trained language models with knowledge injection as knowledge-enhanced pre-trained language models (KEPLMs). These models demonstrate deep understanding and logical reasoning and introduce interpretability. In this survey, we provide a comprehensive overview of KEPLMs in NLP. We first discuss the advancements in pre-trained language models and knowledge representation learning. Then we systematically categorize existing KEPLMs from three different perspectives. Finally, we outline some potential directions of KEPLMs for future research.
32 pages, 15 figures
References in corpus (15)
- Semi-Supervised Classification with Graph Convolutional Networks
- Language Models are Few-Shot Learners
- A Survey on Knowledge Graphs: Representation, Acquisition and Applications
- Pre-trained Models for Natural Language Processing: A Survey
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- Variational Graph Auto-Encoders
- Skip-Thought Vectors
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- REALM: Retrieval-Augmented Language Model Pre-Training
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
- Quasar: Datasets for Question Answering by Search and Reading
- K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
- Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
- Faithful Embeddings for Knowledge Base Queries
- Story Ending Prediction by Transferable BERT