Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries

Abstract

Pretrained language models have been suggested as a possible alternative or complement to structured knowledge bases. However, this emerging LM-as-KB paradigm has so far only been considered in a very limited setting, which only allows handling 21k entities whose single-token name is found in common LM vocabularies. Furthermore, the main benefit of this paradigm, namely querying the KB using a variety of natural language paraphrases, is underexplored so far. Here, we formulate two basic requirements for treating LMs as KBs: (i) the ability to store a large number facts involving a large number of entities and (ii) the ability to query stored facts. We explore three entity representations that allow LMs to represent millions of entities and present a detailed case study on paraphrased querying of world knowledge in LMs, thereby providing a proof-of-concept that language models can indeed serve as knowledge bases.

arXiv commentEACL 2021

References in corpus (9)

• Sequence to Sequence Learning with Neural Networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• TAPAS: Weakly Supervised Table Parsing via Pre-training
• A Neural Knowledge Language Model
• DeepType: Multilingual Entity Linking by Neural Type System Evolution
• Modeling Semantics with Gated Graph Neural Networks for Knowledge Base Question Answering
• Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs

Cited by in corpus (2)

• Language Models As or For Knowledge Bases
• Interpreting Language Models Through Knowledge Graph Extraction