Memory-based Parameter Adaptation

Abstract

Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as they process data, requiring very low learning rates. If the training distribution shifts, the network is slow to adapt, and when it does adapt, it typically performs badly on the training distribution before the shift. Our method, Memory-based Parameter Adaptation, stores examples in memory and then uses a context-based lookup to directly modify the weights of a neural network. Much higher learning rates can be used for this local adaptation, reneging the need for many iterations over similar data before good predictions can be made. As our method is memory-based, it alleviates several shortcomings of neural networks, such as catastrophic forgetting, fast, stable acquisition of new knowledge, learning with an imbalanced class labels, and fast learning during evaluation. We demonstrate this on a range of supervised tasks: large-scale image classification and language modelling.

arXiv commentPublished as a conference paper at ICLR 2018

Cited by in corpus (15)

• Overcoming catastrophic forgetting with hard attention to the task
• Meta-learning with differentiable closed-form solvers
• Meta Continual Learning
• Fast Parametric Learning with Activation Memorization
• Dynamic Relevance Learning for Few-Shot Object Detection
• Overcoming catastrophic forgetting problem by weight consolidation and long-term memory
• Meta-learning algorithms for Few-Shot Computer Vision
• Improved Semantic Role Labeling using Parameterized Neighborhood Memory Adaptation
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Continual Learning in Task-Oriented Dialogue Systems
• Graph-Based Continual Learning
• Dynamically Addressing Unseen Rumor via Continual Learning
• Continual Learning Augmented Investment Decisions
• Improving Sample Efficiency with Normalized RBF Kernels
• Meta-Learning with Variational Bayes