Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference

Abstract

Lack of performance when it comes to continual learning over non-stationary distributions of data remains a major challenge in scaling neural network learning to more human realistic settings. In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off between transfer and interference that can be optimized by enforcing gradient alignment across examples. We then propose a new algorithm, Meta-Experience Replay (MER), that directly exploits this view by combining experience replay with optimization based meta-learning. This method learns parameters that make interference based on future gradients less likely and transfer based on future gradients more likely. We conduct experiments across continual lifelong supervised learning benchmarks and non-stationary reinforcement learning environments demonstrating that our approach consistently outperforms recently proposed baselines for continual learning. Our experiments show that the gap between the performance of MER and baseline algorithms grows both as the environment gets more non-stationary and as the fraction of the total experiences stored gets smaller.

ICLR 2019

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Gradient Episodic Memory for Continual Learning
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Overcoming catastrophic forgetting with hard attention to the task
• A Deeper Look at Experience Replay
• Conditional Computation in Neural Networks for faster models
• Deep Multi-task Representation Learning: A Tensor Factorisation Approach
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Learning Abstract Options
• Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning

Cited by in corpus (25)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Understanding the Role of Training Regimes in Continual Learning
• Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence
• Orthogonal Gradient Descent for Continual Learning
• On-device Training: A First Overview on Existing Systems
• Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning
• Continuum: Simple Management of Complex Continual Learning Scenarios
• Online Continual Learning via the Knowledge Invariant and Spread-out Properties
• MAGR: Manifold-Aligned Graph Regularization for Continual Action Quality Assessment
• Prototypical Q Networks for Automatic Conversational Diagnosis and Few-Shot New Disease Adaption
• Understanding Continual Learning Settings with Data Distribution Drift Analysis
• BI-MAML: Balanced Incremental Approach for Meta Learning
• Latent Domain Learning with Dynamic Residual Adapters
• Learning Invariant Representation for Continual Learning
• GROWN: GRow Only When Necessary for Continual Learning
• Overcoming Catastrophic Interference in Online Reinforcement Learning with Dynamic Self-Organizing Maps
• KSM: Fast Multiple Task Adaption via Kernel-wise Soft Mask Learning
• Continual Reinforcement Learning with Diversity Exploration and Adversarial Self-Correction
• Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension
• Privacy-Preserving Constrained Domain Generalization via Gradient Alignment
• Bilevel Continual Learning
• Lifetime policy reuse and the importance of task capacity
• ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning
• Shared and Private VAEs with Generative Replay for Continual Learning
• Dynamic VAEs with Generative Replay for Continual Zero-shot Learning