Integrating Graph Contextualized Knowledge into Pre-trained Language Models

Abstract

Complex node interactions are common in knowledge graphs, and these interactions also contain rich knowledge information. However, traditional methods usually treat a triple as a training unit during the knowledge representation learning (KRL) procedure, neglecting contextualized information of the nodes in knowledge graphs (KGs). We generalize the modeling object to a very general form, which theoretically supports any subgraph extracted from the knowledge graph, and these subgraphs are fed into a novel transformer-based model to learn the knowledge embeddings. To broaden usage scenarios of knowledge, pre-trained language models are utilized to build a model that incorporates the learned knowledge representations. Experimental results demonstrate that our model achieves the state-of-the-art performance on several medical NLP tasks, and improvement above TransE indicates that our KRL method captures the graph contextualized information effectively.

Findings of EMNLP 2020

References in corpus (5)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network
• ERNIE: Enhanced Representation through Knowledge Integration
• Classifying medical relations in clinical text via convolutional neural networks
• SciBERT: A Pretrained Language Model for Scientific Text

Cited by in corpus (9)

• A Survey on Knowledge Graphs: Representation, Acquisition and Applications
• Pre-trained Models for Natural Language Processing: A Survey
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval
• Large Knowledge Model: Perspectives and Challenges
• Combining pre-trained language models and structured knowledge
• Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models
• K-AID: Enhancing Pre-trained Language Models with Domain Knowledge for Question Answering