Language Models are Open Knowledge Graphs

Abstract

This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pre-trained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.

arXiv comment30 pages, 32 figures, 3 tables

References in corpus (15)

• Semi-Supervised Classification with Graph Convolutional Networks
• Language Models are Few-Shot Learners
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Snorkel: Rapid Training Data Creation with Weak Supervision
• REALM: Retrieval-Augmented Language Model Pre-Training
• Attention is not Explanation
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Language Models as Knowledge Bases?
• How Context Affects Language Models' Factual Predictions
• Hopfield Networks is All You Need
• Generalization through Memorization: Nearest Neighbor Language Models
• Visualizing Attention in Transformer-Based Language Representation Models
• End-to-End Neural Entity Linking

Cited by in corpus (19)

• A Review of Knowledge Graph Completion
• VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
• Towards Zero-Label Language Learning
• Selenite: Scaffolding Online Sensemaking with Comprehensive Overviews Elicited from Large Language Models
• Entity-aware Transformers for Entity Search
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• Language Models As or For Knowledge Bases
• Pre-Trained Models: Past, Present and Future
• Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT
• Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization
• Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
• Patent-KG: Patent Knowledge Graph Use for Engineering Design
• Towards a Universal Continuous Knowledge Base
• Zero-Shot Information Extraction as a Unified Text-to-Triple Translation
• Zero-shot Slot Filling with DPR and RAG
• Predicting Discourse Trees from Transformer-based Neural Summarizers
• TCube: Domain-Agnostic Neural Time-series Narration
• Robust Retrieval Augmented Generation for Zero-shot Slot Filling
• Open Rule Induction