Don't forget, there is more than forgetting: new metrics for Continual Learning

Abstract

Continual learning consists of algorithms that learn from a stream of data/tasks continuously and adaptively thought time, enabling the incremental development of ever more complex knowledge and skills. The lack of consensus in evaluating continual learning algorithms and the almost exclusive focus on forgetting motivate us to propose a more comprehensive set of implementation independent metrics accounting for several factors we believe have practical implications worth considering in the deployment of real AI systems that learn continually: accuracy or performance over time, backward and forward knowledge transfer, memory overhead as well as computational efficiency. Drawing inspiration from the standard Multi-Attribute Value Theory (MAVT) we further propose to fuse these metrics into a single score for ranking purposes and we evaluate our proposal with five continual learning strategies on the iCIFAR-100 continual learning benchmark.

References in corpus (3)

• Continual Learning Through Synaptic Intelligence
• Overcoming catastrophic forgetting with hard attention to the task
• Towards Robust Evaluations of Continual Learning

Cited by in corpus (22)

• Domain-Incremental Continual Learning for Mitigating Bias in Facial Expression and Action Unit Recognition
• Efficient Continual Learning in Neural Networks with Embedding Regularization
• A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
• Orthogonal Gradient Descent for Continual Learning
• Lifelong Continual Learning for Anomaly Detection: New Challenges, Perspectives, and Insights
• Online Continual Learning on Sequences
• When Meta-Learning Meets Online and Continual Learning: A Survey
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning
• Online Continual Learning via the Knowledge Invariant and Spread-out Properties
• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
• Generative Feature Replay For Class-Incremental Learning
• Exploring the Intersection between Neural Architecture Search and Continual Learning
• Evaluating Online Continual Learning with CALM
• ModelCI-e: Enabling Continual Learning in Deep Learning Serving Systems
• CL2R: Compatible Lifelong Learning Representations
• Pseudo-Rehearsal for Continual Learning with Normalizing Flows
• OpenLORIS-Object: A Robotic Vision Dataset and Benchmark for Lifelong Deep Learning
• Bayesian Structure Adaptation for Continual Learning
• A Conceptual Framework for Lifelong Learning
• Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learning
• Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods
• Adaptive Explainable Continual Learning Framework for Regression Problems with Focus on Power Forecasts