A comprehensive, application-oriented study of catastrophic forgetting in DNNs

Abstract

We present a large-scale empirical study of catastrophic forgetting (CF) in modern Deep Neural Network (DNN) models that perform sequential (or: incremental) learning. A new experimental protocol is proposed that enforces typical constraints encountered in application scenarios. As the investigation is empirical, we evaluate CF behavior on the hitherto largest number of visual classification datasets, from each of which we construct a representative number of Sequential Learning Tasks (SLTs) in close alignment to previous works on CF. Our results clearly indicate that there is no model that avoids CF for all investigated datasets and SLTs under application conditions. We conclude with a discussion of potential solutions and workarounds to CF, notably for the EWC and IMM models.

arXiv comment14 pages, 12 + 23 figures, ICLR | 2019 Seventh International Conference on Learning Representations

Cited by in corpus (15)

• A continual learning survey: Defying forgetting in classification tasks
• Compacting, Picking and Growing for Unforgetting Continual Learning
• Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis
• An Investigation of Replay-based Approaches for Continual Learning
• Continual Learning with Adaptive Weights (CLAW)
• A Study of Deep Learning for Network Traffic Data Forecasting
• A Study of Continual Learning Methods for Q-Learning
• Continual Learning with Fully Probabilistic Models
• VIPeR: Visual Incremental Place Recognition with Adaptive Mining and Continual Learning
• Memory Efficient Class-Incremental Learning for Image Classification
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability
• What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning
• A Procedural World Generation Framework for Systematic Evaluation of Continual Learning
• Measuring Information Transfer in Neural Networks