Fast Parametric Learning with Activation Memorization

Abstract

Neural networks trained with backpropagation often struggle to identify classes that have been observed a small number of times. In applications where most class labels are rare, such as language modelling, this can become a performance bottleneck. One potential remedy is to augment the network with a fast-learning non-parametric model which stores recent activations and class labels into an external memory. We explore a simplified architecture where we treat a subset of the model parameters as fast memory stores. This can help retain information over longer time intervals than a traditional memory, and does not require additional space or compute. In the case of image classification, we display faster binding of novel classes on an Omniglot image curriculum task. We also show improved performance for word-based language models on news reports (GigaWord), books (Project Gutenberg) and Wikipedia articles (WikiText-103) --- the latter achieving a state-of-the-art perplexity of 29.2.

References in corpus (15)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Language Modeling with Gated Convolutional Networks
• Matching Networks for One Shot Learning
• Exploring the Limits of Language Modeling
• Pointer Sentinel Mixture Models
• Learning to Remember Rare Events
• Using Fast Weights to Attend to the Recent Past
• Efficient softmax approximation for GPUs
• Deep Meta-Learning: Learning to Learn in the Concept Space
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Search Engine Guided Non-Parametric Neural Machine Translation
• Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling
• Strategies for Training Large Vocabulary Neural Language Models
• Memory-based Parameter Adaptation
• Unbounded cache model for online language modeling with open vocabulary

Cited by in corpus (12)

• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Compressive Transformers for Long-Range Sequence Modelling
• Improving the Gating Mechanism of Recurrent Neural Networks
• Relational recurrent neural networks
• Metalearning with Hebbian Fast Weights
• Universal Language Model Fine-Tuning with Subword Tokenization for Polish
• Progress and Tradeoffs in Neural Language Models
• Adaptive Pruning of Neural Language Models for Mobile Devices
• Do Transformers Need Deep Long-Range Memory
• Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
• MicroNet for Efficient Language Modeling
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future