Knowledge-Aware Language Model Pretraining

Abstract

How much knowledge do pretrained language models hold? Recent research observed that pretrained transformers are adept at modeling semantics but it is unclear to what degree they grasp human knowledge, or how to ensure they do so. In this paper we incorporate knowledge-awareness in language model pretraining without changing the transformer architecture, inserting explicit knowledge layers, or adding external storage of semantic information. Rather, we simply signal the existence of entities to the input of the transformer in pretraining, with an entity-extended tokenizer; and at the output, with an additional entity prediction task. Our experiments show that solely by adding these entity signals in pretraining, significantly more knowledge is packed into the transformer parameters: we observe improved language modeling accuracy, factual correctness in LAMA knowledge probing tasks, and semantics in the hidden representations through edge probing.We also show that our knowledge-aware language model (KALM) can serve as a drop-in replacement for GPT-2 models, significantly improving downstream tasks like zero-shot question-answering with no task-related training.

References in corpus (12)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• The Curious Case of Neural Text Degeneration
• Momentum Contrast for Unsupervised Visual Representation Learning
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Pointer Sentinel Mixture Models
• Word-Entity Duet Representations for Document Ranking
• A Neural Knowledge Language Model
• From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
• Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training

Cited by in corpus (11)

• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• Neural Language Generation: Formulation, Methods, and Evaluation
• Pre-Trained Models: Past, Present and Future
• Pre-training Text-to-Text Transformers for Concept-centric Common Sense
• Combining pre-trained language models and structured knowledge
• KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
• K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretraining
• Billion-scale Pre-trained E-commerce Product Knowledge Graph Model
• Relational World Knowledge Representation in Contextual Language Models: A Review