Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis

Abstract

When an agent acquires new information, ideally it would immediately be capable of using that information to understand its environment. This is not possible using conventional deep neural networks, which suffer from catastrophic forgetting when they are incrementally updated, with new knowledge overwriting established representations. A variety of approaches have been developed that attempt to mitigate catastrophic forgetting in the incremental batch learning scenario, where a model learns from a series of large collections of labeled samples. However, in this setting, inference is only possible after a batch has been accumulated, which prohibits many applications. An alternative paradigm is online learning in a single pass through the training dataset on a resource constrained budget, which is known as streaming learning. Streaming learning has been much less studied in the deep learning community. In streaming learning, an agent learns instances one-by-one and can be tested at any time, rather than only after learning a large batch. Here, we revisit streaming linear discriminant analysis, which has been widely used in the data mining research community. By combining streaming linear discriminant analysis with deep learning, we are able to outperform both incremental batch learning and streaming learning algorithms on both ImageNet ILSVRC-2012 and CORe50, a dataset that involves learning to classify from temporally ordered samples.

arXiv commentTo appear in the IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPR-W) on Continual Learning in Computer Vision (CLVision) 2020

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• How transferable are features in deep neural networks?
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• Shrinkage Algorithms for MMSE Covariance Estimation
• FearNet: Brain-Inspired Model for Incremental Learning
• Measuring Catastrophic Forgetting in Neural Networks
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Towards Robust Evaluations of Continual Learning
• Memory Replay GANs: learning to generate images from new categories without forgetting
• iCub World: Friendly Robots Help Building Good Vision Data-Sets

Cited by in corpus (3)

• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer
• Do We Need Fully Connected Output Layers in Convolutional Networks?