AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training

Abstract

Deep neural networks suffer from catastrophic forgetting when learning multiple knowledge sequentially, and a growing number of approaches have been proposed to mitigate this problem. Some of these methods achieved considerable performance by associating the flat local minima with forgetting mitigation in continual learning. However, they inevitably need (1) tedious hyperparameters tuning, and (2) additional computational cost. To alleviate these problems, in this paper, we propose a simple yet effective optimization method, called AlterSGD, to search for a flat minima in the loss landscape. In AlterSGD, we conduct gradient descent and ascent alternatively when the network tends to converge at each session of learning new knowledge. Moreover, we theoretically prove that such a strategy can encourage the optimization to converge to a flat minima. We verify AlterSGD on continual learning benchmark for semantic segmentation and the empirical results show that we can significantly mitigate the forgetting and outperform the state-of-the-art methods with a large margin under challenging continual learning protocols.

References in corpus (14)

• Overcoming catastrophic forgetting in neural networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• A continual learning survey: Defying forgetting in classification tasks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• Continual Learning Through Synaptic Intelligence
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• OCNet: Object Context Network for Scene Parsing
• Gradient Episodic Memory for Continual Learning
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Reinforced Continual Learning
• Asymmetric Valleys: Beyond Sharp and Flat Local Minima
• Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning
• Efficient Attention Network: Accelerate Attention by Searching Where to Plug