A Strategy for an Uncompromising Incremental Learner

Abstract

Multi-class supervised learning systems require the knowledge of the entire range of labels they predict. Often when learnt incrementally, they suffer from catastrophic forgetting. To avoid this, generous leeways have to be made to the philosophy of incremental learning that either forces a part of the machine to not learn, or to retrain the machine again with a selection of the historic data. While these hacks work to various degrees, they do not adhere to the spirit of incremental learning. In this article, we redefine incremental learning with stringent conditions that do not allow for any undesirable relaxations and assumptions. We design a strategy involving generative models and the distillation of dark knowledge as a means of hallucinating data along with appropriate targets from past distributions. We call this technique, phantom sampling.We show that phantom sampling helps avoid catastrophic forgetting during incremental learning. Using an implementation based on deep neural networks, we demonstrate that phantom sampling dramatically avoids catastrophic forgetting. We apply these strategies to competitive multi-class incremental learning of deep neural networks. Using various benchmark datasets and through our strategy, we demonstrate that strict incremental learning could be achieved. We further put our strategy to test on challenging cases, including cross-domain increments and incrementing on a novel label space. We also propose a trivial extension to unbounded-continual learning and identify potential for future development.

Under review at IEEE Transactions of Neural Networks and Learning Systems

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• FitNets: Hints for Thin Deep Nets
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Less-forgetting Learning in Deep Neural Networks
• Continual Learning in Generative Adversarial Nets
• Active Long Term Memory Networks
• Diving deeper into mentee networks
• Neural Dataset Generality

Cited by in corpus (14)

• Three scenarios for continual learning
• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• Generative replay with feedback connections as a general strategy for continual learning
• Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks
• FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning
• Distillation Techniques for Pseudo-rehearsal Based Incremental Learning
• Incremental Learning In Online Scenario
• A Comparative Study of Calibration Methods for Imbalanced Class Incremental Learning
• OpenLORIS-Object: A Robotic Vision Dataset and Benchmark for Lifelong Deep Learning
• Large Scale Incremental Learning
• Learning to Remember from a Multi-Task Teacher
• Unsupervised Continual Learning Via Pseudo Labels
• Better Knowledge Retention through Metric Learning
• Generative Low-Shot Network Expansion