SATS: Self-Attention Transfer for Continual Semantic Segmentation

Abstract

Continually learning to segment more and more types of image regions is a desired capability for many intelligent systems. However, such continual semantic segmentation suffers from the same catastrophic forgetting issue as in continual classification learning. While multiple knowledge distillation strategies originally for continual classification have been well adapted to continual semantic segmentation, they only consider transferring old knowledge based on the outputs from one or more layers of deep fully convolutional networks. Different from existing solutions, this study proposes to transfer a new type of information relevant to knowledge, i.e. the relationships between elements (Eg. pixels or small local regions) within each image which can capture both within-class and between-class knowledge. The relationship information can be effectively obtained from the self-attention maps in a Transformer-style segmentation model. Considering that pixels belonging to the same class in each image often share similar visual properties, a class-specific region pooling is applied to provide more efficient relationship information for knowledge transfer. Extensive evaluations on multiple public benchmarks support that the proposed self-attention transfer method can further effectively alleviate the catastrophic forgetting issue, and its flexible combination with one or more widely adopted strategies significantly outperforms state-of-the-art solutions.

Published in Pattern Recognition Journal

References in corpus (4)

• Robust Real-World Image Super-Resolution against Adversarial Attacks
• Exploiting Shape Cues for Weakly Supervised Semantic Segmentation
• Test-time Batch Statistics Calibration for Covariate Shift
• PCAMs: Weakly Supervised Semantic Segmentation Using Point Supervision

Cited by in corpus (6)

• AdaER: An Adaptive Experience Replay Approach for Continual Lifelong Learning
• A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
• Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory
• CrackNex: a Few-shot Low-light Crack Segmentation Model Based on Retinex Theory for UAV Inspections
• Learning at a Glance: Towards Interpretable Data-limited Continual Semantic Segmentation via Semantic-Invariance Modelling
• Event-guided Low-light Video Semantic Segmentation