Continual Learning: Forget-free Winning Subnetworks for Video Representations

Abstract

Inspired by the Lottery Ticket Hypothesis (LTH), which highlights the existence of efficient subnetworks within larger, dense networks, a high-performing Winning Subnetwork (WSN) in terms of task performance under appropriate sparsity conditions is considered for various continual learning tasks. It leverages pre-existing weights from dense networks to achieve efficient learning in Task Incremental Learning (TIL) and Task-agnostic Incremental Learning (TaIL) scenarios. In Few-Shot Class Incremental Learning (FSCIL), a variation of WSN referred to as the Soft subnetwork (SoftNet) is designed to prevent overfitting when the data samples are scarce. Furthermore, the sparse reuse of WSN weights is considered for Video Incremental Learning (VIL). The use of Fourier Subneural Operator (FSO) within WSN is considered. It enables compact encoding of videos and identifies reusable subnetworks across varying bandwidths. We have integrated FSO into different architectural frameworks for continual learning, including VIL, TIL, and FSCIL. Our comprehensive experiments demonstrate FSO's effectiveness, significantly improving task performance at various convolutional representational levels. Specifically, FSO enhances higher-layer performance in TIL and FSCIL and lower-layer performance in VIL.

IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI)

References in corpus (12)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Overcoming catastrophic forgetting in neural networks
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Progressive Neural Networks
• Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation
• On Tiny Episodic Memories in Continual Learning
• Overcoming catastrophic forgetting with hard attention to the task
• Continual Learning via Neural Pruning
• New Insights on Reducing Abrupt Representation Change in Online Continual Learning
• Subspace Regularizers for Few-Shot Class Incremental Learning
• Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning
• CNeRV: Content-adaptive Neural Representation for Visual Data