Time-Aware Language Models as Temporal Knowledge Bases

Abstract

Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. But language models (LMs) are trained on snapshots of data collected at a specific moment in time, and this can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum -- those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently "refreshed" as new data arrives, without the need for retraining from scratch.

arXiv commentVersion accepted to TACL

References in corpus (9)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Analysing Lexical Semantic Change with Contextualised Word Representations
• Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change
• Modifying Memories in Transformer Models
• Dynamic Language Models for Continuously Evolving Content
• NewsEmbed: Modeling News through Pre-trained Document Representations
• Editable Neural Networks
• Learning Dynamic Author Representations with Temporal Language Models
• Temporal Reasoning on Implicit Events from Distant Supervision

Cited by in corpus (6)

• On the Opportunities and Risks of Foundation Models
• Towards Continual Knowledge Learning of Language Models
• A Dataset for Answering Time-Sensitive Questions
• Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs
• Time Masking for Temporal Language Models
• Temporal Image Caption Retrieval Competition -- Description and Results