Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics

Abstract

A central challenge in developing versatile machine learning systems is catastrophic forgetting: a model trained on tasks in sequence will suffer significant performance drops on earlier tasks. Despite the ubiquity of catastrophic forgetting, there is limited understanding of the underlying process and its causes. In this paper, we address this important knowledge gap, investigating how forgetting affects representations in neural network models. Through representational analysis techniques, we find that deeper layers are disproportionately the source of forgetting. Supporting this, a study of methods to mitigate forgetting illustrates that they act to stabilize deeper layers. These insights enable the development of an analytic argument and empirical picture relating the degree of forgetting to representational similarity between tasks. Consistent with this picture, we observe maximal forgetting occurs for task sequences with intermediate similarity. We perform empirical studies on the standard split CIFAR-10 setup and also introduce a novel CIFAR-100 based task approximating realistic input distribution shift.

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Overcoming catastrophic forgetting in neural networks
• mixup: Beyond Empirical Risk Minimization
• Deep Residual Learning for Image Recognition
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• A mathematical theory of semantic development in deep neural networks
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Measuring Catastrophic Forgetting in Neural Networks
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Are All Layers Created Equal?
• Continual Learning via Neural Pruning
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• A Survey of Deep Learning for Scientific Discovery
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• Toward Understanding Catastrophic Forgetting in Continual Learning

Cited by in corpus (7)

• A General Language Assistant as a Laboratory for Alignment
• A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix
• Linear Mode Connectivity in Multitask and Continual Learning
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• When Do Curricula Work?
• Center Loss Regularization for Continual Learning
• Continual learning under domain transfer with sparse synaptic bursting