Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer
arXiv:2101.09536
Abstract
Rehearsal is a critical component for class-incremental continual learning, yet it requires a substantial memory budget. Our work investigates whether we can significantly reduce this memory budget by leveraging unlabeled data from an agent's environment in a realistic and challenging continual learning paradigm. Specifically, we explore and formalize a novel semi-supervised continual learning (SSCL) setting, where labeled data is scarce yet non-i.i.d. unlabeled data from the agent's environment is plentiful. Importantly, data distributions in the SSCL setting are realistic and therefore reflect object class correlations between, and among, the labeled and unlabeled data distributions. We show that a strategy built on pseudo-labeling, consistency regularization, Out-of-Distribution (OoD) detection, and knowledge distillation reduces forgetting in this setting. Our approach, DistillMatch, increases performance over the state-of-the-art by no less than 8.7% average task accuracy and up to 54.5% average task accuracy in SSCL CIFAR-100 experiments. Moreover, we demonstrate that DistillMatch can save up to 0.23 stored images per processed unlabeled image compared to the next best method which only saves 0.08. Our results suggest that focusing on realistic correlated distributions is a significantly new perspective, which accentuates the importance of leveraging the world's structure as a continual learning strategy.
arXiv commentAccepted by the 2021 International Joint Conference on Neural Networks (IJCNN 2021)
References in corpus (23)
- Overcoming catastrophic forgetting in neural networks
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Semi-Supervised Learning with Deep Generative Models
- Continual Learning Through Synaptic Intelligence
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
- Efficient Lifelong Learning with A-GEM
- Three scenarios for continual learning
- Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
- On Tiny Episodic Memories in Continual Learning
- FearNet: Brain-Inspired Model for Incremental Learning
- Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
- Measuring Catastrophic Forgetting in Neural Networks
- Deep Generative Dual Memory Network for Continual Learning
- B-CNN: Branch Convolutional Neural Network for Hierarchical Classification
- A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
- Functional Regularisation for Continual Learning with Gaussian Processes
- Learning to Continually Learn
- Task Agnostic Continual Learning via Meta Learning
- Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis
- Generative Feature Replay For Class-Incremental Learning
- Manifold Graph with Learned Prototypes for Semi-Supervised Image Classification