Improving Neural Language Models with a Continuous Cache

Abstract

We propose an extension to neural network language models to adapt their prediction to the recent history. Our model is a simplified version of memory augmented networks, which stores past hidden activations as memory and accesses them through a dot product with the current hidden activation. This mechanism is very efficient and scales to very large memory sizes. We also draw a link between the use of external memory in neural network and cache models used with count based language models. We demonstrate on several language model datasets that our approach performs significantly better than recent memory augmented networks.

Submitted to ICLR 2017

Cited by in corpus (39)

• Learning and Evaluating General Linguistic Intelligence
• Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code
• Trellis Networks for Sequence Modeling
• Paradigm Shift in Natural Language Processing
• Multiplicative LSTM for sequence modelling
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• Dynamic Evaluation of Neural Sequence Models
• Generalization through Memorization: Nearest Neighbor Language Models
• DeLighT: Deep and Light-weight Transformer
• Augmenting Self-attention with Persistent Memory
• A Survey on Neural Network Language Models
• Deep Independently Recurrent Neural Network (IndRNN)
• Dynamic Evaluation of Transformer Language Models
• Explaining and Improving Model Behavior with k Nearest Neighbor Representations
• Nearest Neighbor Machine Translation
• Augmenting Transformers with KNN-Based Composite Memory for Dialogue
• Time-aware Large Kernel Convolutions
• DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
• Low-Resource Language Modelling of South African Languages
• Mutual Information Scaling and Expressive Power of Sequence Models
• Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
• Retrieval Augmentation Reduces Hallucination in Conversation
• Improved Semantic Role Labeling using Parameterized Neighborhood Memory Adaptation
• Memory Augmented Neural Model for Incremental Session-based Recommendation
• Character n-gram Embeddings to Improve RNN Language Models
• Adaptive Semiparametric Language Models
• Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation
• Non-local Recurrent Neural Memory for Supervised Sequence Modeling
• Persistence pays off: Paying Attention to What the LSTM Gating Mechanism Persists
• $\infty$-former: Infinite Memory Transformer
• Memory and attention in deep learning
• Identifying and Exploiting Structures for Reliable Deep Learning
• Representation Memorization for Fast Learning New Knowledge without Forgetting
• Towards Continual Entity Learning in Language Models for Conversational Agents
• Single Training Dimension Selection for Word Embedding with PCA
• Thick-Net: Parallel Network Structure for Sequential Modeling
• MicroNet for Efficient Language Modeling
• Evaluating Computational Language Models with Scaling Properties of Natural Language
• Neural Language Modeling With Implicit Cache Pointers