CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability
arXiv:2110.03331
Abstract
What is the state of the art in continual machine learning? Although a natural question for predominant static benchmarks, the notion to train systems in a lifelong manner entails a plethora of additional challenges with respect to set-up and evaluation. The latter have recently sparked a growing amount of critiques on prominent algorithm-centric perspectives and evaluation protocols being too narrow, resulting in several attempts at constructing guidelines in favor of specific desiderata or arguing against the validity of prevalent assumptions. In this work, we depart from this mindset and argue that the goal of a precise formulation of desiderata is an ill-posed one, as diverse applications may always warrant distinct scenarios. Instead, we introduce the Continual Learning EValuation Assessment Compass: the CLEVA-Compass. The compass provides the visual means to both identify how approaches are practically reported and how works can simultaneously be contextualized in the broader literature landscape. In addition to promoting compact specification in the spirit of recent replication trends, it thus provides an intuitive chart to understand the priorities of individual systems, where they resemble each other, and what elements are missing towards a fair comparison.
arXiv commentInternational Conference on Learning Representations (ICLR) 2022
References in corpus (15)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- A continual learning survey: Defying forgetting in classification tasks
- Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
- Efficient Lifelong Learning with A-GEM
- Three scenarios for continual learning
- Deep Reinforcement Learning that Matters
- The Natural Language Decathlon: Multitask Learning as Question Answering
- Measuring Catastrophic Forgetting in Neural Networks
- Towards Robust Evaluations of Continual Learning
- On The Power of Curriculum Learning in Training Deep Networks
- A comprehensive, application-oriented study of catastrophic forgetting in DNNs
- Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition
- Understanding Continual Learning Settings with Data Distribution Drift Analysis
- Sequoia: A Software Framework to Unify Continual Learning Research