A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
arXiv:2310.14277 · doi:10.1109/TPAMI.2024.3446949
Abstract
Continual learning, also known as incremental learning or life-long learning, stands at the forefront of deep learning and AI systems. It breaks through the obstacle of one-way training on close sets and enables continuous adaptive learning on open-set conditions. In the recent decade, continual learning has been explored and applied in multiple fields especially in computer vision covering classification, detection and segmentation tasks. Continual semantic segmentation (CSS), of which the dense prediction peculiarity makes it a challenging, intricate and burgeoning task. In this paper, we present a review of CSS, committing to building a comprehensive survey on problem formulations, primary challenges, universal datasets, neoteric theories and multifarious applications. Concretely, we begin by elucidating the problem definitions and primary challenges. Based on an in-depth investigation of relevant approaches, we sort out and categorize current CSS models into two main branches including data-replay and data-free sets. In each branch, the corresponding approaches are similarity-based clustered and thoroughly analyzed, following qualitative comparison and quantitative reproductions on relevant datasets. Besides, we also introduce four CSS specialities with diverse application scenarios and development tendencies. Furthermore, we develop a benchmark for CSS encompassing representative references, evaluation results and reproductions, which is available at~\url{https://github.com/YBIO/SurveyCSS}. We hope this survey can serve as a reference-worthy and stimulating contribution to the advancement of the life-long learning field, while also providing valuable perspectives for related fields.
arXiv comment20 pages, 11 figures
References in corpus (39)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distilling the Knowledge in a Neural Network
- Overcoming catastrophic forgetting in neural networks
- Rethinking Atrous Convolution for Semantic Image Segmentation
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
- A continual learning survey: Defying forgetting in classification tasks
- Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
- DINOv2: Learning Robust Visual Features without Supervision
- Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks
- Class-Incremental Learning: A Survey
- A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
- Brain-like associative learning using a nanoscale non-volatile phase change synaptic device array
- Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications
- Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data
- Don't forget, there is more than forgetting: new metrics for Continual Learning
- Double Similarity Distillation for Semantic Image Segmentation
- Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation
- DAugNet: Unsupervised, Multi-source, Multi-target, and Life-long Domain Adaptation for Semantic Segmentation of Satellite Images
- End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent
- Triple Memory Networks: a Brain-Inspired Method for Continual Learning
- Semantic-SAM: Segment and Recognize Anything at Any Granularity
- SATS: Self-Attention Transfer for Continual Semantic Segmentation
- Domain Adaptation Meets Zero-Shot Learning: An Annotation-Efficient Approach to Multi-Modality Medical Image Segmentation
- Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation
- Domain-incremental Cardiac Image Segmentation with Style-oriented Replay and Domain-sensitive Feature Whitening
- Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory
- Multi-view Incremental Segmentation of 3D Point Clouds for Mobile Robots
- Architecture Matters in Continual Learning
- Birds of A Feather Flock Together: Category-Divergence Guidance for Domain Adaptive Segmentation
- Incremental Learning for Multi-organ Segmentation with Partially Labeled Datasets
- Continual Coarse-to-Fine Domain Adaptation in Semantic Segmentation
- Continual Adaptation of Semantic Segmentation using Complementary 2D-3D Data Representations
- Learning at a Glance: Towards Interpretable Data-limited Continual Semantic Segmentation via Semantic-Invariance Modelling
- CoDEPS: Online Continual Learning for Depth Estimation and Panoptic Segmentation
- Panoptic Perception: A Novel Task and Fine-grained Dataset for Universal Remote Sensing Image Interpretation
- What is Wrong with Continual Learning in Medical Image Segmentation?
- Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation
- DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation
- Is Continual Learning Ready for Real-world Challenges?