Integrating Graph Contextualized Knowledge into Pre-trained Language Models

Abstract

Complex node interactions are common in knowledge graphs, and these interactions also contain rich knowledge information. However, traditional methods usually treat a triple as a training unit during the knowledge representation learning (KRL) procedure, neglecting contextualized information of the nodes in knowledge graphs (KGs). We generalize the modeling object to a very general form, which theoretically supports any subgraph extracted from the knowledge graph, and these subgraphs are fed into a novel transformer-based model to learn the knowledge embeddings. To broaden usage scenarios of knowledge, pre-trained language models are utilized to build a model that incorporates the learned knowledge representations. Experimental results demonstrate that our model achieves the state-of-the-art performance on several medical NLP tasks, and improvement above TransE indicates that our KRL method captures the graph contextualized information effectively.

arXiv commentFindings of EMNLP 2020

References in corpus (5)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network
• ERNIE: Enhanced Representation through Knowledge Integration
• Classifying medical relations in clinical text via convolutional neural networks
• SciBERT: A Pretrained Language Model for Scientific Text

Cited by in corpus (9)

• A Survey on Knowledge Graphs: Representation, Acquisition and Applications
• Pre-trained Models for Natural Language Processing: A Survey
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval
• Large Knowledge Model: Perspectives and Challenges
• Combining pre-trained language models and structured knowledge
• Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models
• K-AID: Enhancing Pre-trained Language Models with Domain Knowledge for Question Answering