Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation

Abstract

Deep learning approaches are nowadays ubiquitously used to tackle computer vision tasks such as semantic segmentation, requiring large datasets and substantial computational power. Continual learning for semantic segmentation (CSS) is an emerging trend that consists in updating an old model by sequentially adding new classes. However, continual learning methods are usually prone to catastrophic forgetting. This issue is further aggravated in CSS where, at each step, old classes from previous iterations are collapsed into the background. In this paper, we propose Local POD, a multi-scale pooling distillation scheme that preserves long- and short-range spatial relationships at feature level. Furthermore, we design an entropy-based pseudo-labelling of the background w.r.t. classes predicted by the old model to deal with background shift and avoid catastrophic forgetting of the old classes. Finally, we introduce a novel rehearsal method that is particularly suited for segmentation. Our approach, called PLOP, significantly outperforms state-of-the-art methods in existing CSS scenarios, as well as in newly proposed challenging benchmarks.

arXiv commentUnder review at IEEE TPAMI, journal extension of arXiv:2011.11390

References in corpus (26)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• OCNet: Object Context Network for Scene Parsing
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Hierarchical Multi-Scale Attention for Semantic Segmentation
• On Tiny Episodic Memories in Continual Learning
• Measuring Catastrophic Forgetting in Neural Networks
• Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data
• Continual Learning via Neural Pruning
• Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
• Learn the new, keep the old: Extending pretrained models with new anatomy and images
• M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning
• DisCoRL: Continual Reinforcement Learning via Policy Distillation
• Continuum: Simple Management of Complex Continual Learning Scenarios
• Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigence
• Understanding Continual Learning Settings with Data Distribution Drift Analysis
• ObjectAug: Object-level Data Augmentation for Semantic Image Segmentation