Towards Robust Evaluations of Continual Learning

Abstract

Experiments used in current continual learning research do not faithfully assess fundamental challenges of learning continually. Instead of assessing performance on challenging and representative experiment designs, recent research has focused on increased dataset difficulty, while still using flawed experiment set-ups. We examine standard evaluations and show why these evaluations make some continual learning approaches look better than they are. We introduce desiderata for continual learning evaluations and explain why their absence creates misleading comparisons. Based on our desiderata we then propose new experiment designs which we demonstrate with various continual learning approaches and datasets. Our analysis calls for a reprioritization of research effort by the community.

References in corpus (18)

• Overcoming catastrophic forgetting in neural networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• How transferable are features in deep neural networks?
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Weight Uncertainty in Neural Networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Efficient Lifelong Learning with A-GEM
• Variational Dropout and the Local Reparameterization Trick
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Less-forgetting Learning in Deep Neural Networks
• On Quadratic Penalties in Elastic Weight Consolidation
• A Unifying Bayesian View of Continual Learning

Cited by in corpus (77)

• A continual learning survey: Defying forgetting in classification tasks
• Three scenarios for continual learning
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Generative replay with feedback connections as a general strategy for continual learning
• Continual Learning for Recurrent Neural Networks: an Empirical Evaluation
• Implicit Generation and Generalization in Energy-Based Models
• Don't forget, there is more than forgetting: new metrics for Continual Learning
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• Understanding the Role of Training Regimes in Continual Learning
• Functional Regularisation for Continual Learning with Gaussian Processes
• Coresets via Bilevel Optimization for Continual Learning and Streaming
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• Gradient based sample selection for online continual learning
• Online Continual Learning with Maximally Interfered Retrieval
• Task Agnostic Continual Learning Using Online Variational Bayes
• Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis
• SpaceNet: Make Free Space For Continual Learning
• AdaER: An Adaptive Experience Replay Approach for Continual Lifelong Learning
• A Unifying Bayesian View of Continual Learning
• CLeaR: An Adaptive Continual Learning Framework for Regression Tasks
• Orthogonal Gradient Descent for Continual Learning
• Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition
• Continual Semi-Supervised Learning through Contrastive Interpolation Consistency
• Class-incremental Learning with Pre-allocated Fixed Classifiers
• Toward Understanding Catastrophic Forgetting in Continual Learning
• Improved Schemes for Episodic Memory-based Lifelong Learning
• Learning to Continuously Optimize Wireless Resource In Episodically Dynamic Environment
• REMIND Your Neural Network to Prevent Catastrophic Forgetting
• Continual learning of longitudinal health records
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning
• A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems
• Meta-Learning with Sparse Experience Replay for Lifelong Language Learning
• A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix
• Online Class-Incremental Continual Learning with Adversarial Shapley Value
• Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning
• Continual Learning on the Edge with TensorFlow Lite
• Continual Learning in Neural Networks
• Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis
• Linear Mode Connectivity in Multitask and Continual Learning
• Imbalanced Continual Learning with Partitioning Reservoir Sampling
• Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors
• Robust Importance Sampling for Error Estimation in the Context of Optimal Bayesian Transfer Learning
• Energy-Based Models for Continual Learning
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain
• Unifying Regularisation Methods for Continual Learning
• Localizing Catastrophic Forgetting in Neural Networks
• Learning Invariant Representation for Continual Learning
• Look At Me, No Replay! SurpriseNet: Anomaly Detection Inspired Class Incremental Learning
• Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data
• Sequoia: A Software Framework to Unify Continual Learning Research
• Continual Learning Using Multi-view Task Conditional Neural Networks
• Continual Learning with Fully Probabilistic Models
• Posterior Meta-Replay for Continual Learning
• Neuromodulated Neural Architectures with Local Error Signals for Memory-Constrained Online Continual Learning
• Reviewing continual learning from the perspective of human-level intelligence
• Efficient Data-Parallel Continual Learning with Asynchronous Distributed Rehearsal Buffers
• Rethinking Experience Replay: a Bag of Tricks for Continual Learning
• Continual Learning from the Perspective of Compression
• Single-Net Continual Learning with Progressive Segmented Training (PST)
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
• CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability
• Algorithmic insights on continual learning from fruit flies
• Variational Auto-Regressive Gaussian Processes for Continual Learning
• CLOPS: Continual Learning of Physiological Signals
• Closed-Loop Memory GAN for Continual Learning
• How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role of Interpolation
• Continual Learning in Deep Networks: an Analysis of the Last Layer
• Learning Neural Models for Natural Language Processing in the Face of Distributional Shift
• ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning
• Online Continual Learning in Image Classification: An Empirical Survey
• Class Incremental Online Streaming Learning
• OvA-INN: Continual Learning with Invertible Neural Networks
• Adaptive Explainable Continual Learning Framework for Regression Problems with Focus on Power Forecasts
• Does the Adam Optimizer Exacerbate Catastrophic Forgetting?
• FoCL: Feature-Oriented Continual Learning for Generative Models
• Toward industrial use of continual learning : new metrics proposal for class incremental learning