Tuned Compositional Feature Replays for Efficient Stream Learning

Abstract

Our brains extract durable, generalizable knowledge from transient experiences of the world. Artificial neural networks come nowhere close to this ability. When tasked with learning to classify objects by training on non-repeating video frames in temporal order (online stream learning), models that learn well from shuffled datasets catastrophically forget old knowledge upon learning new stimuli. We propose a new continual learning algorithm, Compositional Replay Using Memory Blocks (CRUMB), which mitigates forgetting by replaying feature maps reconstructed by combining generic parts. CRUMB concatenates trainable and re-usable "memory block" vectors to compositionally reconstruct feature map tensors in convolutional neural networks. Storing the indices of memory blocks used to reconstruct new stimuli enables memories of the stimuli to be replayed during later tasks. This reconstruction mechanism also primes the neural network to minimize catastrophic forgetting by biasing it towards attending to information about object shapes more than information about image textures, and stabilizes the network during stream learning by providing a shared feature-level basis for all training examples. These properties allow CRUMB to outperform an otherwise identical algorithm that stores and replays raw images, while occupying only 3.6% as much memory. We stress-tested CRUMB alongside 13 competing methods on 7 challenging datasets. To address the limited number of existing online stream learning datasets, we introduce 2 new benchmarks by adapting existing datasets for stream learning. With only 3.7-4.1% as much memory and 15-43% as much runtime, CRUMB mitigates catastrophic forgetting more effectively than the state-of-the-art. Our code is available at https://github.com/MorganBDT/crumb.git.

arXiv commentCopyright 2023 IEEE. The journal version of this article is hosted at https://ieeexplore.ieee.org/document/10373937 and https://klab.tch.harvard.edu/publications/PDFs/gk8019.pdf

References in corpus (19)

• Overcoming catastrophic forgetting in neural networks
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• A continual learning survey: Defying forgetting in classification tasks
• Continual Learning Through Synaptic Intelligence
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• Progress & Compress: A scalable framework for continual learning
• Variational Continual Learning
• Mnemonics Training: Multi-Class Incremental Learning without Forgetting
• Adaptive Aggregation Networks for Class-Incremental Learning
• Continual Learning via Neural Pruning
• Understanding the Role of Training Regimes in Continual Learning
• Overcoming Long-term Catastrophic Forgetting through Adversarial Neural Pruning and Synaptic Consolidation
• Pseudo-Recursal: Solving the Catastrophic Forgetting Problem in Deep Neural Networks
• Prototype-Guided Memory Replay for Continual Learning
• Super-Samples from Kernel Herding
• Continual Learning with Adaptive Weights (CLAW)
• The Toybox Dataset of Egocentric Visual Object Transformations
• Few-Shot Self Reminder to Overcome Catastrophic Forgetting