REMIND Your Neural Network to Prevent Catastrophic Forgetting

Abstract

People learn throughout life. However, incrementally updating conventional neural networks leads to catastrophic forgetting. A common remedy is replay, which is inspired by how the brain consolidates memory. Replay involves fine-tuning a network on a mixture of new and old instances. While there is neuroscientific evidence that the brain replays compressed memories, existing methods for convolutional networks replay raw images. Here, we propose REMIND, a brain-inspired approach that enables efficient replay with compressed representations. REMIND is trained in an online manner, meaning it learns one example at a time, which is closer to how humans learn. Under the same constraints, REMIND outperforms other methods for incremental class learning on the ImageNet ILSVRC-2012 dataset. We probe REMIND's robustness to data ordering schemes known to induce catastrophic forgetting. We demonstrate REMIND's generality by pioneering online learning for Visual Question Answering (VQA).

arXiv commentTo appear in the European Conference on Computer Vision (ECCV-2020)

References in corpus (18)

• Overcoming catastrophic forgetting in neural networks
• How transferable are features in deep neural networks?
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• VQA: Visual Question Answering
• Continual Learning Through Synaptic Intelligence
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• Manifold Mixup: Better Representations by Interpolating Hidden States
• FearNet: Brain-Inspired Model for Incremental Learning
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• Measuring Catastrophic Forgetting in Neural Networks
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Towards Robust Evaluations of Continual Learning
• Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
• FigureQA: An Annotated Figure Dataset for Visual Reasoning
• On transfer learning using a MAC model variant
• DeeSIL: Deep-Shallow Incremental Learning

Cited by in corpus (4)

• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• Meta-Consolidation for Continual Learning
• Semantic Segmentation in Multiple Adverse Weather Conditions with Domain Knowledge Retention
• Class Incremental Online Streaming Learning