Generalization through Memorization: Nearest Neighbor Language Models

Abstract

We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this augmentation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our $k$NN-LM achieves a new state-of-the-art perplexity of 15.79 - a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.

ICLR 2020

References in corpus (9)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Pointer Sentinel Mixture Models
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• MBT: A Memory-Based Part of Speech Tagger-Generator
• Improving Neural Language Models with a Continuous Cache
• Dynamic Evaluation of Transformer Language Models
• Retrieval-Augmented Convolutional Neural Networks for Improved Robustness against Adversarial Examples
• Lightweight Adaptive Mixture of Neural and N-gram Language Models

Cited by in corpus (15)

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• REALM: Retrieval-Augmented Language Model Pre-Training
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Pre-training via Paraphrasing
• Language Models are Open Knowledge Graphs
• BASE Layers: Simplifying Training of Large, Sparse Models
• Cross-lingual Retrieval for Iterative Self-Supervised Training
• Addressing Some Limitations of Transformers with Feedback Memory
• GNN-LM: Language Modeling based on Global Contexts via GNN
• Retrieval-Augmented Transformer-XL for Close-Domain Dialog Generation
• Taking Notes on the Fly Helps BERT Pre-training
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
• Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
• Ember: No-Code Context Enrichment via Similarity-Based Keyless Joins
• Leveraging Reinforcement Learning for evaluating Robustness of KNN Search Algorithms