KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs
arXiv:2109.04223
Abstract
Incorporating factual knowledge into pre-trained language models (PLM) such as BERT is an emerging trend in recent NLP studies. However, most of the existing methods combine the external knowledge integration module with a modified pre-training loss and re-implement the pre-training process on the large-scale corpus. Re-pretraining these models is usually resource-consuming, and difficult to adapt to another domain with a different knowledge graph (KG). Besides, those works either cannot embed knowledge context dynamically according to textual context or struggle with the knowledge ambiguity issue. In this paper, we propose a novel knowledge-aware language model framework based on fine-tuning process, which equips PLM with a unified knowledge-enhanced text graph that contains both text and multi-relational sub-graphs extracted from KG. We design a hierarchical relational-graph-based message passing mechanism, which can allow the representations of injected KG and text to mutually update each other and can dynamically select ambiguous mentioned entities that share the same text. Our empirical results show that our model can efficiently incorporate world knowledge from KGs into existing language models such as BERT, and achieve significant improvement on the machine reading comprehension (MRC) task compared with other knowledge-enhanced models.
arXiv commentICLR 2022 on DLG4NLP
References in corpus (14)
- Semi-Supervised Classification with Graph Convolutional Networks
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- NLTK: The Natural Language Toolkit
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
- K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
- K-BERT: Enabling Language Representation with Knowledge Graph
- Modeling Relational Data with Graph Convolutional Networks
- LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
- Graph Star Net for Generalized Multi-Task Learning
- CoLAKE: Contextualized Language and Knowledge Embedding