Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory

Abstract

As a front-burner problem in incremental learning, class incremental semantic segmentation (CISS) is plagued by catastrophic forgetting and semantic drift. Although recent methods have utilized knowledge distillation to transfer knowledge from the old model, they are still unable to avoid pixel confusion, which results in severe misclassification after incremental steps due to the lack of annotations for past and future classes. Meanwhile data-replay-based approaches suffer from storage burdens and privacy concerns. In this paper, we propose to address CISS without exemplar memory and resolve catastrophic forgetting as well as semantic drift synchronously. We present Inherit with Distillation and Evolve with Contrast (IDEC), which consists of a Dense Knowledge Distillation on all Aspects (DADA) manner and an Asymmetric Region-wise Contrastive Learning (ARCL) module. Driven by the devised dynamic class-specific pseudo-labelling strategy, DADA distils intermediate-layer features and output-logits collaboratively with more emphasis on semantic-invariant knowledge inheritance. ARCL implements region-wise contrastive learning in the latent space to resolve semantic drift among known classes, current classes, and unknown classes. We demonstrate the effectiveness of our method on multiple CISS tasks by state-of-the-art performance, including Pascal VOC 2012, ADE20K and ISPRS datasets. Our method also shows superior anti-forgetting ability, particularly in multi-step CISS tasks.

References in corpus (26)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• A Simple Framework for Contrastive Learning of Visual Representations
• In Defense of the Triplet Loss for Person Re-Identification
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• A continual learning survey: Defying forgetting in classification tasks
• Unsupervised Representation Learning by Predicting Image Rotations
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks
• Born Again Neural Networks
• Experience Replay for Continual Learning
• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data
• Continual Lifelong Learning in Natural Language Processing: A Survey
• Episodic Memory in Lifelong Language Learning
• Double Similarity Distillation for Semantic Image Segmentation
• Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation
• Online Continual Learning with Maximally Interfered Retrieval
• SATS: Self-Attention Transfer for Continual Semantic Segmentation
• M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning
• Recent Advances of Continual Learning in Computer Vision: An Overview
• Channel Distillation: Channel-Wise Attention for Knowledge Distillation
• Multi-Task Incremental Learning for Object Detection
• Embedded Self-Distillation in Compact Multi-Branch Ensemble Network for Remote Sensing Scene Classification

Cited by in corpus (3)

• A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
• Learning at a Glance: Towards Interpretable Data-limited Continual Semantic Segmentation via Semantic-Invariance Modelling
• Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images