KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

Abstract

Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE models cannot take full advantage of the abundant textual information. In this paper, we propose a unified model for Knowledge Embedding and Pre-trained LanguagE Representation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs. In KEPLER, we encode textual entity descriptions with a PLM as their embeddings, and then jointly optimize the KE and language modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, and also works remarkably well as an inductive KE model on KG link prediction. Furthermore, for pre-training and evaluating KEPLER, we construct Wikidata5M, a large-scale KG dataset with aligned entity descriptions, and benchmark state-of-the-art KE methods on it. It shall serve as a new KE benchmark and facilitate the research on large KG, inductive KE, and KG with text. The source code can be obtained from https://github.com/THU-KEG/KEPLER.

arXiv commentAccepted to TACL

References in corpus (8)

• Distributed Representations of Words and Phrases and their Compositionality
• Prototypical Networks for Few-shot Learning
• Inductive Representation Learning on Large Graphs
• RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• GraphVite: A High-Performance CPU-GPU Hybrid System for Node Embedding
• Joint Representation Learning of Text and Knowledge for Knowledge Graph Completion

Cited by in corpus (8)

• Pre-trained Models for Natural Language Processing: A Survey
• Compute Trends Across Three Eras of Machine Learning
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce
• KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
• Does Knowledge Help General NLU? An Empirical Study
• Embedding Ontologies via Incorporating Extensional and Intensional Knowledge
• Billion-scale Pre-trained E-commerce Product Knowledge Graph Model