Learning to Remember from a Multi-Task Teacher
arXiv:1910.04650
Abstract
Recent studies on catastrophic forgetting during sequential learning typically focus on fixing the accuracy of the predictions for a previously learned task. In this paper we argue that the outputs of neural networks are subject to rapid changes when learning a new data distribution, and networks that appear to "forget" everything still contain useful representation towards previous tasks. Instead of enforcing the output accuracy to stay the same, we propose to reduce the effect of catastrophic forgetting on the representation level, as the output layer can be quickly recovered later with a small number of examples. Towards this goal, we propose an experimental setup that measures the amount of representational forgetting, and develop a novel meta-learning algorithm to overcome this issue. The proposed meta-learner produces weight updates of a sequential learning network, mimicking a multi-task teacher network's representation. We show that our meta-learner can improve its learned representations on new tasks, while maintaining a good representation for old tasks.
References in corpus (14)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Distilling the Knowledge in a Neural Network
- Overcoming catastrophic forgetting in neural networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Continual Learning Through Synaptic Intelligence
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- Overcoming Catastrophic Forgetting by Incremental Moment Matching
- Generative replay with feedback connections as a general strategy for continual learning
- Understanding and correcting pathologies in the training of learned optimizers
- A Strategy for an Uncompromising Incremental Learner
- Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity
- Reviving and Improving Recurrent Back-Propagation
- Overcoming Multi-Model Forgetting