Condensed Composite Memory Continual Learning
arXiv:2102.09890
Abstract
Deep Neural Networks (DNNs) suffer from a rapid decrease in performance when trained on a sequence of tasks where only data of the most recent task is available. This phenomenon, known as catastrophic forgetting, prevents DNNs from accumulating knowledge over time. Overcoming catastrophic forgetting and enabling continual learning is of great interest since it would enable the application of DNNs in settings where unrestricted access to all the training data at any time is not always possible, e.g. due to storage limitations or legal issues. While many recently proposed methods for continual learning use some training examples for rehearsal, their performance strongly depends on the number of stored examples. In order to improve performance of rehearsal for continual learning, especially for a small number of stored examples, we propose a novel way of learning a small set of synthetic examples which capture the essence of a complete dataset. Instead of directly learning these synthetic examples, we learn a weighted combination of shared components for each example that enables a significant increase in memory efficiency. We demonstrate the performance of our method on commonly used datasets and compare it to recently proposed related methods and baselines.
arXiv commentPaper accepted for publication at IJCNN2021
References in corpus (11)
- Overcoming catastrophic forgetting in neural networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Prototypical Networks for Few-shot Learning
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Continual Learning Through Synaptic Intelligence
- Three scenarios for continual learning
- Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
- Generative replay with feedback connections as a general strategy for continual learning
- Dataset Condensation with Gradient Matching
- Coresets via Bilevel Optimization for Continual Learning and Streaming
- Learning to Continually Learn