Mind the Gap: Assessing Temporal Generalization in Neural Language Models

Abstract

Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contrasts with the current static language modelling paradigm, which trains and evaluates models on utterances from overlapping time periods. Despite impressive recent progress, we demonstrate that Transformer-XL language models perform worse in the realistic setup of predicting future utterances from beyond their training period, and that model performance becomes increasingly worse with time. We find that, while increasing model size alone -- a key driver behind recent progress -- does not solve this problem, having models that continually update their knowledge with new information can indeed mitigate this performance degradation over time. Hence, given the compilation of ever-larger language modelling datasets, combined with the growing list of language-model-based NLP applications that require up-to-date factual knowledge about the world, we argue that now is the right time to rethink the static way in which we currently train and evaluate our language models, and develop adaptive language models that can remain up-to-date with respect to our ever-changing and non-stationary world. We publicly release our dynamic, streaming language modelling benchmarks for WMT and arXiv to facilitate language model evaluation that takes temporal dynamics into account.

arXiv commentTo appear as a Spotlight at NeurIPS 2021

References in corpus (16)

• Overcoming catastrophic forgetting in neural networks
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Scaling Laws for Neural Language Models
• Frustratingly Easy Domain Adaptation
• Generating Long Sequences with Sparse Transformers
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Reformer: The Efficient Transformer
• Continuous Time Dynamic Topic Models
• Carbon Emissions and Large Neural Network Training
• LAMOL: LAnguage MOdeling for Lifelong Language Learning
• Modifying Memories in Transformer Models
• Dynamic Evaluation of Transformer Language Models
• The Global Anchor Method for Quantifying Linguistic Shifts and Domain Adaptation
• Editable Neural Networks
• Evaluating Online Continual Learning with CALM

Cited by in corpus (5)

• State-of-the-art generalisation research in NLP: A taxonomy and review
• Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models
• Emergent Language: A Survey and Taxonomy
• Clarify: Improving Model Robustness With Natural Language Corrections
• Boosting classification reliability of NLP transformer models in the long run